From af7b90e7f119606b27d207aefe29c207245a569d Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Fri, 21 Aug 2026 12:46:27 -0400 Subject: [PATCH 1/8] Draft release notes for 3.1 --- docs/sources/tempo/release-notes/v3-1.md | 244 +++++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 docs/sources/tempo/release-notes/v3-1.md diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md new file mode 100644 index 00000000000..96bb0a11344 --- /dev/null +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -0,0 +1,244 @@ +--- +title: Version 3.1 release notes +menuTitle: V3.1 +description: Release notes for Grafana Tempo 3.1 +weight: 650 +--- + +# Version 3.1 release notes + + + + + + +The Tempo team is pleased to announce the release of Grafana Tempo 3.1. + +This release gives you: + +- [Redaction by query](#redaction-by-query): Redact traces that match a TraceQL query instead of enumerating IDs, so a sensitive-data incident doesn't require hunting down every affected trace. +- [TraceQL metrics extrapolation and arithmetic](#traceql-metrics-extrapolation-and-arithmetic): Extrapolate counts from head-sampled traces so rates reflect actual traffic, and combine aggregations with arithmetic to calculate ratios like error rate in a single query. +- [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. +- [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. +- [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. + +These release notes highlight the most important features and bug fixes. +For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). + +## Redaction by query + +When sensitive data accidentally lands in traces, you often need to prove it's gone on a compliance deadline, not wait for retention to expire. +Tempo 3.0 introduced [trace redaction](/docs/tempo//release-notes/v3-0/#trace-redaction), but you had to supply an explicit list of trace IDs. +Finding every affected ID first is slow at tenant scale, and a missed ID leaves the data in storage. + +Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. [[PR 7663](https://github.com/grafana/tempo/pull/7663)] + +```bash +tempo-cli redact --tenant= --query '{ resource.service.name = "checkout" && span.http.status_code = 500 }' --dry-run +``` + +The `--query` and `--trace-id` flags are mutually exclusive -- exactly one is required. +Run with `--dry-run` first; the CLI does not rewrite blocks, and match counts are reported on the `tempo_backend_scheduler_redaction_traces_found_total` metric, not printed by the command. +Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for the query syntax and constraints. + +### Backend scheduler redaction reliability + +Query-based selection only helps if the redaction actually finishes. +This release closes several correctness gaps in the backend scheduler's redaction pipeline that could leave a job looking stuck, skip a compacted block, or pause compaction during a dry run: + +- A completed redaction batch now enters a short quiescence period before removal, so a block compacted in the window right after the last redaction job finishes still gets covered by a rescan instead of escaping redaction. [[PR 7695](https://github.com/grafana/tempo/pull/7695)] +- Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path where the body's `tenant_id` field was previously trusted. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] +- A dry-run redaction no longer disables the tenant's compaction and retention or arms a rescan; those now only happen for a redaction in apply mode, which actually rewrites blocks. [[PR 7700](https://github.com/grafana/tempo/pull/7700)] +- A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could leave a tenant's redaction looking perpetually in progress and block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] +- Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log make it observable when a redaction job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] +- A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode, so you can preview a redaction's blast radius before it runs, alongside a new Redaction row on the Backend Work dashboard. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] + +The backend scheduler also got several scalability fixes so job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) + +Refer to [Compaction](/docs/tempo//reference-tempo-architecture/components/compaction/) for the redaction job lifecycle, metrics, and Backend Work dashboard. + +## TraceQL metrics extrapolation and arithmetic + +If you use head-based sampling to control costs, +metrics derived from those traces reflect only the sampled fraction, not actual traffic. +Request rates, error counts, and throughput all undercount by the sampling factor. +At 10% sampling, `{ } | rate()` reports 50 req/s when real traffic is 500 req/s. +Getting accurate numbers meant a parallel metrics pipeline, +the ingest-time metrics-generator, or multiplying by hand in every dashboard. + +Tempo 3.1 lets TraceQL metrics extrapolate those counts from the W3C `tracestate` sampling probability on the read path. +The numbers come from traces you already stored, so you don't need extra series or a second pipeline. +Opt in per query with the experimental `with(extrapolate=true)` hint. +When a span carries an OpenTelemetry probability threshold in its tracestate, +it contributes `1 / sampling_probability` to `rate`, `count_over_time`, `sum_over_time`, `avg_over_time`, `histogram_over_time`, `quantile_over_time`, and `compare`. +That matches the metrics-generator's existing per-span multiplier behavior. +`min_over_time` and `max_over_time` are unaffected, and the hint requires vParquet4 or later. +[[PR 7452](https://github.com/grafana/tempo/pull/7452)] + +TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`). +You can combine aggregations in a single query instead of running two queries and stitching them client-side. +Dashboards stay simpler, and the time buckets line up. +Divide two rates to get an error rate, or scale a result by a constant. +(PRs [#6866](https://github.com/grafana/tempo/pull/6866), [#7199](https://github.com/grafana/tempo/pull/7199), [#7409](https://github.com/grafana/tempo/pull/7409)) + +This query on head-sampled data returns an error rate that reflects real traffic: + +```traceql +({status=error} | rate()) / ({} | rate()) with(extrapolate=true) +``` + +Refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/) for the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the extrapolation hint. + +The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false`, or per-query with the hint `with(spanonly_fetch=false)`, if you hit a regression. Refer to [Faster read path](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path). [[PR 7179](https://github.com/grafana/tempo/pull/7179)] + +This release also fixes several TraceQL correctness issues: instant metrics queries now correctly reuse the per-block job results cache [[PR 7602](https://github.com/grafana/tempo/pull/7602)]; the vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `{ trace:rootService="..." } | rate()`, `span:childCount`, and array operations, or for metrics queries on event and link intrinsics [[PR 7508](https://github.com/grafana/tempo/pull/7508), [PR 7533](https://github.com/grafana/tempo/pull/7533)]; and `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. [[PR 7170](https://github.com/grafana/tempo/pull/7170)] + +## Metrics-generator and service graph improvements + +If you use the metrics-generator, you get RED metrics and a service map from traces you already ingest. +The generator processes every span at ingest, so CPU, restart time, and dropped edges all affect the series you alert on. + +Service graphs get a new opt-in `traces_service_graph_connection_info` presence gauge, +powered by new `service-graphs-*` subprocessors, +for detecting service topology under heavy sampling, +visualized in a new `tempo-service-graph.json` dashboard. [[PR 7202](https://github.com/grafana/tempo/pull/7202)] +An edge used to require both the client and server span from the same trace; +at low sample rates, one side is often missing and the connection never appears. +The new subprocessors detect service-to-service connections from individual spans, +so the map stays more complete even when most traces are dropped. +Refer to [Connection information metric](/docs/tempo//metrics-from-traces/service_graphs/#connection-information-metric) to enable the subprocessor and query the gauge. + +Additional service graph improvements: + +- Service graphs now recognize the `db.system.name` attribute (the OpenTelemetry v1.30.0 rename of `db.system`) for database node detection and virtual node naming. + Both attributes remain supported, so database nodes still appear after you upgrade to OpenTelemetry SDKs that emit the renamed attribute. [[PR 7697](https://github.com/grafana/tempo/pull/7697), [documentation](/docs/tempo//metrics-from-traces/service_graphs/#database-name-attributes)] +- Expired service-graph edges are now labeled with the unmatched span kind, so you can tell which side of a connection was missing instead of an edge disappearing with no explanation. [[PR 7709](https://github.com/grafana/tempo/pull/7709), [documentation](/docs/tempo//troubleshooting/metrics-generator/#expired-edges)] + +The span-metrics and service-graphs processors were both reworked to build series labels through the metrics-generator registry's pooled, borrowed-label path, cutting per-span and per-edge CPU and allocations. +At high ingest rates, that per-span cost is what makes the generator expensive to run. +Metric names, labels, and values are unchanged. (PRs [#7584](https://github.com/grafana/tempo/pull/7584), [#7587](https://github.com/grafana/tempo/pull/7587)) +As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior. + +A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. Refer to [Metrics-generator](/docs/tempo//configuration/#metrics-generator). [[PR 7611](https://github.com/grafana/tempo/pull/7611)] +A restart or scale-up no longer spends time processing spans that would never become metrics. +Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing `tempo_ingest_group_partition_lag` series after a partition handoff. [[PR 7665](https://github.com/grafana/tempo/pull/7665)] +That series is how you tell whether the generator is keeping up; +after a handoff it used to keep growing, which made health dashboards hard to trust. + +## Span pruning for trace-by-id v2 + +The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. [[PR 7566](https://github.com/grafana/tempo/pull/7566)] + +An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don't explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren't re-pruned. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override controls this independently of the cluster-wide default. [[PR 7693](https://github.com/grafana/tempo/pull/7693)] + +To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant `span_pruning_awareness` override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. [[PR 7532](https://github.com/grafana/tempo/pull/7532)] + +Refer to [Query V2](/docs/tempo//api_docs/#query-v2) for the request parameters and [Query-frontend](/docs/tempo//configuration/#query-frontend) for the cluster-wide and per-tenant settings. + +## Kafka ingestion improvements + +In microservices mode, Tempo uses Kafka as a durable write-ahead log: +traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. +Until 3.1, the Tempo Kafka client couldn't authenticate, encrypt in transit, prefer a local rack, or choose a compression codec. +If your cluster required SASL or TLS, you couldn't use this path at all. + +Tempo 3.1 adds SASL authentication (`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, or `AWS_MSK_IAM`) and TLS/mutual-TLS, +so you can connect to authenticated, encrypted Kafka, including Amazon MSK. +A `client_rack` option enables rack-aware fetching (KIP-392). +Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, +which cuts cross-zone data transfer cost on high-throughput pipelines. +An `ingest.kafka.producer_compression` option overrides the producer's compression codec, +which Azure Event Hubs requires because it only supports `gzip`. +(PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691)) + +Community contributors independently unblocked each of those gaps: +SASL and TLS from [@heytrav](https://github.com/heytrav), +rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), +and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). +Thanks to all three for this work. + +Refer to [Configure authentication and TLS](/docs/tempo//set-up-for-tracing/setup-tempo/configure-kafka/#configure-authentication-and-tls) for SASL and TLS, and [Ingest](/docs/tempo//configuration/#ingest) for `client_rack` and `producer_compression`. + +The distributor also now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click "Forget" after a non-graceful Pod termination in most cases. Refer to [Tune the consistent hash rings](/docs/tempo//operations/manage-advanced-systems/consistent_hash_ring/#distributor). [[PR 7098](https://github.com/grafana/tempo/pull/7098)] + +## Trace diff and trace summary (experimental) + +Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)] + +By default, the diff reports a compact summary alongside the full patch, up to 64 KiB; larger patches report that the patch was omitted rather than truncating it silently (`trace-summary-v0-composed`). [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] + +`tempo-cli` also gets trace diff support for local work: `trace diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] + +Refer to [Trace diff](/docs/tempo//api_docs/#trace-diff) for the HTTP API, [Compare traces](/docs/tempo//api_docs/mcp-server/#compare-traces) for the MCP tool, and [Experimental trace diff](/docs/tempo//operations/tempo_cli/#experimental-trace-diff) for the CLI. + +## Features and enhancements + +The most important remaining features and enhancements in Tempo 3.1 are highlighted below. + +- Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo//operations/verify-container-images/)) +- KEDA-based autoscaling is now available for live-store via a Prometheus trigger on expected bytes held, and for the metrics-generator in Jsonnet. New top-level `autoscaling_prometheus_url` and `autoscaling_prometheus_tenant` fields configure the Prometheus source; set `autoscaling_prometheus_tenant` when the source is a multi-tenant system such as Grafana Mimir. (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) +- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes on all query paths, and a `tempo_querier_backend_processing_duration_seconds` histogram for time the querier spends processing backend blocks (excluding live-store data). Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [documentation](/docs/tempo//reference-tempo-architecture/components/querier/#key-metrics)) +- The trace-by-id v2 endpoint adds a first pass of filtering support, with `q` (TraceQL filter) and `keep_hierarchy` query parameters to return only matching spans. [[PR 7483](https://github.com/grafana/tempo/pull/7483), [documentation](/docs/tempo//api_docs/#query-v2)] +- The Tempo MCP server's documentation tools were expanded and refreshed: a new `docs-config` tool and `docs://config/overview` / `docs://config/reference` resources serve the configuration reference (generated from the default configuration, so it stays in sync with the code), and the TraceQL and metrics documentation served by the MCP server was updated to match current capabilities. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo//api_docs/mcp-server/#available-tools)) +- Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group (on a 2.4 GB block, a high-cardinality scan dropped from 67ms to 9ms); `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner, freeing space for active blocks. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) +- The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options for tuning connection behavior. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) +- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology from `traces_service_graph_connection_info`, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [documentation](/docs/tempo//operations/monitor/#dashboards)) +- A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] +- Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] + +## Upgrade considerations + +When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps. + +### Redis client rewrite + +The experimental Redis cache client has been completely rewritten: Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. If you don't use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] + +### Query sharding: `blocks_per_shard` takes precedence over `query_shards` + +Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] + +### Metrics queries with identical start and end timestamps are now rejected + +As part of a fix to the metrics job results cache, a metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] + +### Default gRPC streaming packet size reduced + +The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MB to 1 MB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. Refer to [Response larger than the max](/docs/tempo//troubleshooting/querying/response-too-large/). [[PR 7615](https://github.com/grafana/tempo/pull/7615)] + +### Memcached idle connections stay open by default + +Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to a positive value. Refer to [Memcached cache connection defaults](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults). [[PR 7671](https://github.com/grafana/tempo/pull/7671)] + +### Docker image tags + +Release artifacts no longer publish mutable Docker image tags; images are published only to immutable GAR repositories, and example configs are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] + +### Go version upgrade + +Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. + +## Security fixes + +- Updated Go to 1.26.5 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to fix CVE-2026-39822, CVE-2026-42504, CVE-2026-27145, CVE-2026-42505, CVE-2026-42507, CVE-2026-46600, and CVE-2026-56852. [[PR 7641](https://github.com/grafana/tempo/pull/7641)] +- Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] + +## Bug fixes + +For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). + +- Bound per-trace slice preallocation during distributor rebatching. [[PR 7288](https://github.com/grafana/tempo/pull/7288)] +- Return a retryable status for transient errors writing to Kafka. [[PR 7506](https://github.com/grafana/tempo/pull/7506)] +- Cache empty tag-value results per block so a block with no matching values isn't re-scanned on every query. [[PR 7617](https://github.com/grafana/tempo/pull/7617)] +- Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. [[PR 7155](https://github.com/grafana/tempo/pull/7155)] +- Stop silently dropping a block's tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [[PR 7610](https://github.com/grafana/tempo/pull/7610)] +- Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn't page at `+Inf%` when no records have been produced yet. [[PR 7715](https://github.com/grafana/tempo/pull/7715)] +- Fix a data race in the query frontend when dispatching request batches to queriers with `max_batch_size > 1`. [[PR 7664](https://github.com/grafana/tempo/pull/7664)] +- Reject TraceQL queries larger than `max_query_expression_size_bytes` before parsing them. [[PR 7245](https://github.com/grafana/tempo/pull/7245), [documentation](/docs/tempo//configuration/#cap-the-maximum-query-length)] +- Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. [[PR 7271](https://github.com/grafana/tempo/pull/7271)] +- Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. [[PR 7662](https://github.com/grafana/tempo/pull/7662)] +- Fix a rare cache collision between instant and range metrics queries. [[PR 7290](https://github.com/grafana/tempo/pull/7290)] +- Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it's now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] +- Fix `=nil` queries missing some values in TraceQL. [[PR 7345](https://github.com/grafana/tempo/pull/7345)] +- Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). [[PR 7660](https://github.com/grafana/tempo/pull/7660)] +- Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [[PR 7609](https://github.com/grafana/tempo/pull/7609)] From cb2c36455f51348d59569495ba5ffb7bf3eb4377 Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Tue, 25 Aug 2026 14:11:42 -0400 Subject: [PATCH 2/8] Add vP5 as default and other missing items --- docs/sources/tempo/release-notes/v3-1.md | 67 +++++++++++++++++-- .../set-up-for-tracing/setup-tempo/upgrade.md | 30 +++++++++ 2 files changed, 90 insertions(+), 7 deletions(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index 96bb0a11344..73366207480 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -21,6 +21,7 @@ This release gives you: - [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. - [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. - [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. +- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 and vParquet3 blocks still read, with no migration. These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). @@ -38,9 +39,21 @@ tempo-cli redact --tenant= --query '{ resource.service.name = "checko ``` The `--query` and `--trace-id` flags are mutually exclusive -- exactly one is required. -Run with `--dry-run` first; the CLI does not rewrite blocks, and match counts are reported on the `tempo_backend_scheduler_redaction_traces_found_total` metric, not printed by the command. +Run with `--dry-run` first; in dry-run mode no blocks are rewritten, and match counts are reported on the `tempo_backend_scheduler_redaction_traces_found_total` metric, not printed by the command. Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for the query syntax and constraints. +You can also bound a query-based redaction with `--start` and `--end` +(`now`, a relative offset such as `now-7d`, or an RFC3339 timestamp). +Both bounds are required, and they can't be combined with `--trace-id`. +A windowed redaction only scans blocks that overlap the range, +so a large tenant can be redacted in slices instead of holding compaction off for the whole run. +[[PR 7702](https://github.com/grafana/tempo/pull/7702)] + +{{< admonition type="caution" >}} +Don't submit a windowed redaction until every scheduler and worker is on Tempo 3.1. +An older worker ignores the window and redacts every query match in the block, with no error. +{{< /admonition >}} + ### Backend scheduler redaction reliability Query-based selection only helps if the redaction actually finishes. @@ -52,6 +65,9 @@ This release closes several correctness gaps in the backend scheduler's redactio - A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could leave a tenant's redaction looking perpetually in progress and block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] - Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log make it observable when a redaction job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] - A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode, so you can preview a redaction's blast radius before it runs, alongside a new Redaction row on the Backend Work dashboard. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] +- A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth (jobs enqueued and not yet assigned), so you can tell whether a redaction needs more workers. + Unlike `jobs_active`, which is capped by worker count, this is the signal that can drive scale-up. + Refer to [Key metrics](/docs/tempo//reference-tempo-architecture/components/compaction/#key-metrics). [[PR 7772](https://github.com/grafana/tempo/pull/7772)] The backend scheduler also got several scalability fixes so job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) @@ -169,6 +185,10 @@ By default, the diff reports a compact summary alongside the full patch, up to 6 `tempo-cli` also gets trace diff support for local work: `trace diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] +The MCP server also exposes a `trace-diff` tool for complete-trace comparisons, +with a compact composed summary by default and patches up to 64 KiB. +[[PR 7785](https://github.com/grafana/tempo/pull/7785)] + Refer to [Trace diff](/docs/tempo//api_docs/#trace-diff) for the HTTP API, [Compare traces](/docs/tempo//api_docs/mcp-server/#compare-traces) for the MCP tool, and [Experimental trace diff](/docs/tempo//operations/tempo_cli/#experimental-trace-diff) for the CLI. ## Features and enhancements @@ -176,27 +196,58 @@ Refer to [Trace diff](/docs/tempo//api_docs/#trace-diff) for the The most important remaining features and enhancements in Tempo 3.1 are highlighted below. - Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo//operations/verify-container-images/)) -- KEDA-based autoscaling is now available for live-store via a Prometheus trigger on expected bytes held, and for the metrics-generator in Jsonnet. New top-level `autoscaling_prometheus_url` and `autoscaling_prometheus_tenant` fields configure the Prometheus source; set `autoscaling_prometheus_tenant` when the source is a multi-tenant system such as Grafana Mimir. (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) -- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes on all query paths, and a `tempo_querier_backend_processing_duration_seconds` histogram for time the querier spends processing backend blocks (excluding live-store data). Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [documentation](/docs/tempo//reference-tempo-architecture/components/querier/#key-metrics)) -- The trace-by-id v2 endpoint adds a first pass of filtering support, with `q` (TraceQL filter) and `keep_hierarchy` query parameters to return only matching spans. [[PR 7483](https://github.com/grafana/tempo/pull/7483), [documentation](/docs/tempo//api_docs/#query-v2)] +- KEDA-based autoscaling is now available for live-store via a Prometheus trigger on expected bytes held, and for the metrics-generator in Jsonnet. + New top-level `autoscaling_prometheus_url` and `autoscaling_prometheus_tenant` fields configure the Prometheus source; set `autoscaling_prometheus_tenant` when the source is a multi-tenant system such as Grafana Mimir. + The live-store trigger uses `AverageValue`; using `Value` scales the cluster to `maxReplicas` regardless of load. + (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) +- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes on all query paths, a `tempo_querier_backend_processing_duration_seconds` histogram for time the querier spends processing backend blocks (excluding live-store data), and stats metrics on additional querier methods. + Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). + (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [#7571](https://github.com/grafana/tempo/pull/7571), [#7568](https://github.com/grafana/tempo/pull/7568), [documentation](/docs/tempo//reference-tempo-architecture/components/querier/#key-metrics)) +- The trace-by-id v2 endpoint adds a first pass of filtering support, with `q` (TraceQL filter) and `keep_hierarchy` query parameters to return only matching spans. + `match_depth` (default `0`) bounds how many descendant hops to keep, and `ancestor_depth` (default `-1`, only with `keep_hierarchy=true`) bounds ancestor hops. + `tempo-cli query trace-id` adds matching `--match-depth` and `--ancestor-depth` flags. + [[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo//api_docs/#query-v2)] - The Tempo MCP server's documentation tools were expanded and refreshed: a new `docs-config` tool and `docs://config/overview` / `docs://config/reference` resources serve the configuration reference (generated from the default configuration, so it stays in sync with the code), and the TraceQL and metrics documentation served by the MCP server was updated to match current capabilities. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo//api_docs/mcp-server/#available-tools)) - Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group (on a 2.4 GB block, a high-cardinality scan dropped from 67ms to 9ms); `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner, freeing space for active blocks. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) - The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options for tuning connection behavior. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) -- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology from `traces_service_graph_connection_info`, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [documentation](/docs/tempo//operations/monitor/#dashboards)) +- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology from `traces_service_graph_connection_info`, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. + The Service Topology dashboard queries `traces_service_graph_request_total`. + (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo//operations/monitor/#dashboards)) - A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] - Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] +- An opt-in `engine_bytes_tracking` override reports protobuf-like span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`. + This isn't parquet bytes inspected, and it's disabled by default. + [[PR 7689](https://github.com/grafana/tempo/pull/7689), [documentation](/docs/tempo//configuration/#overrides)] +- New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, so you can tune `max_input_blocks`. + [[PR 7773](https://github.com/grafana/tempo/pull/7773)] ## Upgrade considerations When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps. +### Default block format is now vParquet5 + +Tempo 3.1 writes new blocks in vParquet5. +Existing vParquet4 and vParquet3 blocks still read, and no data migration is required. +To keep writing vParquet4, set the block version explicitly: + +```yaml +storage: + trace: + block: + version: vParquet4 +``` + +Refer to [Default block format is now vParquet5](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) and [Apache Parquet block format](/docs/tempo//configuration/parquet/). +[[PR 7775](https://github.com/grafana/tempo/pull/7775)] + ### Redis client rewrite The experimental Redis cache client has been completely rewritten: Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. If you don't use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] ### Query sharding: `blocks_per_shard` takes precedence over `query_shards` -Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] +Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] ### Metrics queries with identical start and end timestamps are now rejected @@ -208,7 +259,7 @@ The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MB to ### Memcached idle connections stay open by default -Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to a positive value. Refer to [Memcached cache connection defaults](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults). [[PR 7671](https://github.com/grafana/tempo/pull/7671)] +Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. Refer to [Memcached cache connection defaults](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults). [[PR 7671](https://github.com/grafana/tempo/pull/7671)] ### Docker image tags @@ -242,3 +293,5 @@ For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/t - Fix `=nil` queries missing some values in TraceQL. [[PR 7345](https://github.com/grafana/tempo/pull/7345)] - Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). [[PR 7660](https://github.com/grafana/tempo/pull/7660)] - Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [[PR 7609](https://github.com/grafana/tempo/pull/7609)] +- Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [[PR 7787](https://github.com/grafana/tempo/pull/7787)] +- Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn't compacted or polled before it's complete. [[PR 7786](https://github.com/grafana/tempo/pull/7786)] diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md index c92f87eb8ce..e979db5aac2 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md @@ -31,6 +31,23 @@ You can check your configuration options using the [`status` API endpoint](https ## Upgrade to Tempo 3.1 +### Default block format is now vParquet5 + +Tempo 3.1 writes new blocks in vParquet5. +Existing vParquet4 and vParquet3 blocks still read, and no data migration is required. +[[PR 7775](https://github.com/grafana/tempo/pull/7775)] + +To keep writing vParquet4, set the block version explicitly: + +```yaml +storage: + trace: + block: + version: vParquet4 +``` + +Refer to [Apache Parquet block format](/docs/tempo//configuration/parquet/) for version details. + ### Redis cache configuration changes Tempo upgrades the Redis cache client to [`github.com/redis/go-redis/v9`](https://github.com/redis/go-redis) and reworks the cache configuration. Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. [[PR 7337](https://github.com/grafana/tempo/pull/7337)] @@ -134,6 +151,19 @@ cache: min_idle_conns: 0 # Minimum idle connections to maintain in the pool. ``` +### Trace by ID query sharding + +Trace-by-ID lookups now shard dynamically based on the number of blocks in the blocklist rather than using a fixed shard count. A new `blocks_per_shard` option defaults to `30` and takes precedence over the deprecated `query_shards` setting. [[PR 7105](https://github.com/grafana/tempo/pull/7105)] + +To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`: + +```yaml +query_frontend: + trace_by_id: + blocks_per_shard: 0 + query_shards: 50 # previous default +``` + ### Memcached cache connection defaults Tempo changes the default connection behavior of the memcached cache client to keep connection pools warm across request bursts. If you do not use the memcached cache, no action is needed. [[PR 7671](https://github.com/grafana/tempo/pull/7671)] From fb6001e8d2258e08f8ab7d5c4e8d549d794fcda2 Mon Sep 17 00:00:00 2001 From: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> Date: Thu, 27 Aug 2026 02:25:29 -0400 Subject: [PATCH 3/8] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/sources/tempo/release-notes/v3-1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index 73366207480..b550283e22d 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -38,7 +38,7 @@ Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you c tempo-cli redact --tenant= --query '{ resource.service.name = "checkout" && span.http.status_code = 500 }' --dry-run ``` -The `--query` and `--trace-id` flags are mutually exclusive -- exactly one is required. +The `--query` and `--trace-id` flags are mutually exclusive; exactly one is required. Run with `--dry-run` first; in dry-run mode no blocks are rewritten, and match counts are reported on the `tempo_backend_scheduler_redaction_traces_found_total` metric, not printed by the command. Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for the query syntax and constraints. From fe2332b86453d82979e006756ca3dd48a78d818d Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Wed, 9 Sep 2026 09:00:43 -0400 Subject: [PATCH 4/8] Add enforced vP3 deprecation --- docs/sources/tempo/configuration/parquet.md | 13 +++++----- docs/sources/tempo/release-notes/v3-1.md | 14 +++++++++-- .../set-up-for-tracing/setup-tempo/upgrade.md | 24 +++++++++++++++++-- 3 files changed, 41 insertions(+), 10 deletions(-) diff --git a/docs/sources/tempo/configuration/parquet.md b/docs/sources/tempo/configuration/parquet.md index 53da72211a6..d75701a02e1 100644 --- a/docs/sources/tempo/configuration/parquet.md +++ b/docs/sources/tempo/configuration/parquet.md @@ -12,7 +12,7 @@ This format is required for tags-based search as well as [TraceQL](../../traceql The columnar block format improves search performance and enables an ecosystem of tools, including [Tempo CLI](https://grafana.com/docs/tempo//operations/tempo_cli/#analyse-blocks), to access the underlying trace data. Starting in Tempo 3.1, Tempo writes new blocks in `vParquet5` by default. -Existing `vParquet4` and `vParquet3` blocks remain readable. +Existing `vParquet4` and `vParquet3` blocks remain readable, but `vParquet3` writes and compaction are no longer supported. No data migration is required. ## Considerations @@ -25,14 +25,15 @@ As soon as a block format version is enabled, Tempo starts writing data in that ## Block format versions -{{< admonition type="warning" >}} +Only Parquet-based formats are supported. + +### Removed and deprecated block formats + The `v2` block format has been removed in Tempo 3.0. -`vParquet3` is deprecated. + +`vParquet3` deprecation is enforced starting in Tempo 3.1: Tempo refuses to start if configured to write `vParquet3` blocks, and existing `vParquet3` blocks are no longer compacted. Tempo 3.x still reads existing `vParquet3` blocks. Write new blocks in `vParquet5` (default) or `vParquet4`. -{{< /admonition >}} - -Only Parquet-based formats are supported. ### vParquet5 diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index b550283e22d..dcc83271800 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -21,7 +21,7 @@ This release gives you: - [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. - [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. - [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. -- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 and vParquet3 blocks still read, with no migration. +- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks still read, with no migration. These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). @@ -228,7 +228,7 @@ When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgr ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. -Existing vParquet4 and vParquet3 blocks still read, and no data migration is required. +Existing vParquet4 blocks still read, and no data migration is required. To keep writing vParquet4, set the block version explicitly: ```yaml @@ -241,6 +241,16 @@ storage: Refer to [Default block format is now vParquet5](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) and [Apache Parquet block format](/docs/tempo//configuration/parquet/). [[PR 7775](https://github.com/grafana/tempo/pull/7775)] +### vParquet3 deprecation enforced + +vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. +Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. +Reads of existing vParquet3 blocks are unaffected. +[[PR 7858](https://github.com/grafana/tempo/pull/7858)] + +If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading. +Refer to [vParquet3 deprecation enforced](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) and [Apache Parquet block format](/docs/tempo//configuration/parquet/). + ### Redis client rewrite The experimental Redis cache client has been completely rewritten: Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. If you don't use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md index e979db5aac2..46d59bf8d8a 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md @@ -34,7 +34,7 @@ You can check your configuration options using the [`status` API endpoint](https ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. -Existing vParquet4 and vParquet3 blocks still read, and no data migration is required. +Existing vParquet4 blocks still read, and no data migration is required. [[PR 7775](https://github.com/grafana/tempo/pull/7775)] To keep writing vParquet4, set the block version explicitly: @@ -48,11 +48,31 @@ storage: Refer to [Apache Parquet block format](/docs/tempo//configuration/parquet/) for version details. +### vParquet3 deprecation enforced + +vParquet3 deprecation is now enforced in Tempo 3.1 [[PR 7858](https://github.com/grafana/tempo/pull/7858)]. +This means that: + +- Tempo refuses to start if configured to write vParquet3 blocks. +- Existing vParquet3 blocks are no longer compacted. +- Reads of existing vParquet3 blocks are unaffected. + +If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading: + +```yaml +storage: + trace: + block: + version: vParquet5 +``` + +Refer to [Apache Parquet block format](/docs/tempo//configuration/parquet/) for version details. + ### Redis cache configuration changes Tempo upgrades the Redis cache client to [`github.com/redis/go-redis/v9`](https://github.com/redis/go-redis) and reworks the cache configuration. Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. [[PR 7337](https://github.com/grafana/tempo/pull/7337)] -If you do not use the Redis cache, no action is needed. +If you don't use the Redis cache, no action is needed. #### Opt single-node Redis into the single-node client From 36ce0598a34df1c18dec7e419b039aa89c808af5 Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Wed, 9 Sep 2026 10:36:25 -0400 Subject: [PATCH 5/8] docs: Address Copilot review comments on 3.1 notes Fix grammar, remove the duplicate query-sharding section, and point the release notes at the remaining upgrade heading. --- docs/sources/tempo/release-notes/v3-1.md | 4 ++-- .../set-up-for-tracing/setup-tempo/upgrade.md | 15 +-------------- 2 files changed, 3 insertions(+), 16 deletions(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index dcc83271800..7c094babb47 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -228,7 +228,7 @@ When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgr ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. -Existing vParquet4 blocks still read, and no data migration is required. +Existing vParquet4 blocks are still readable, and no data migration is required. To keep writing vParquet4, set the block version explicitly: ```yaml @@ -257,7 +257,7 @@ The experimental Redis cache client has been completely rewritten: Redis Cluster ### Query sharding: `blocks_per_shard` takes precedence over `query_shards` -Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] +Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding now scales with block count](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] ### Metrics queries with identical start and end timestamps are now rejected diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md index 46d59bf8d8a..57279023b58 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md @@ -34,7 +34,7 @@ You can check your configuration options using the [`status` API endpoint](https ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. -Existing vParquet4 blocks still read, and no data migration is required. +Existing vParquet4 blocks are still readable, and no data migration is required. [[PR 7775](https://github.com/grafana/tempo/pull/7775)] To keep writing vParquet4, set the block version explicitly: @@ -171,19 +171,6 @@ cache: min_idle_conns: 0 # Minimum idle connections to maintain in the pool. ``` -### Trace by ID query sharding - -Trace-by-ID lookups now shard dynamically based on the number of blocks in the blocklist rather than using a fixed shard count. A new `blocks_per_shard` option defaults to `30` and takes precedence over the deprecated `query_shards` setting. [[PR 7105](https://github.com/grafana/tempo/pull/7105)] - -To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`: - -```yaml -query_frontend: - trace_by_id: - blocks_per_shard: 0 - query_shards: 50 # previous default -``` - ### Memcached cache connection defaults Tempo changes the default connection behavior of the memcached cache client to keep connection pools warm across request bursts. If you do not use the memcached cache, no action is needed. [[PR 7671](https://github.com/grafana/tempo/pull/7671)] From e0b9df69ba67115b9a28cce7128900a25bf907da Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Fri, 11 Sep 2026 17:38:16 -0400 Subject: [PATCH 6/8] Update rel notes to address Javi's feedback --- docs/sources/tempo/release-notes/v3-1.md | 223 +++++++++--------- .../setup-tempo/command-line-flags.md | 7 + 2 files changed, 116 insertions(+), 114 deletions(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index 7c094babb47..70ca1e56a98 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -21,6 +21,7 @@ This release gives you: - [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. - [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. - [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. +- [Trace diff and trace summary (experimental)](#trace-diff-and-trace-summary-experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, `tempo-cli`, and the MCP server. - [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks still read, with no migration. These release notes highlight the most important features and bug fixes. @@ -28,73 +29,33 @@ For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/t ## Redaction by query -When sensitive data accidentally lands in traces, you often need to prove it's gone on a compliance deadline, not wait for retention to expire. Tempo 3.0 introduced [trace redaction](/docs/tempo//release-notes/v3-0/#trace-redaction), but you had to supply an explicit list of trace IDs. -Finding every affected ID first is slow at tenant scale, and a missed ID leaves the data in storage. - Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. [[PR 7663](https://github.com/grafana/tempo/pull/7663)] -```bash -tempo-cli redact --tenant= --query '{ resource.service.name = "checkout" && span.http.status_code = 500 }' --dry-run -``` - -The `--query` and `--trace-id` flags are mutually exclusive; exactly one is required. -Run with `--dry-run` first; in dry-run mode no blocks are rewritten, and match counts are reported on the `tempo_backend_scheduler_redaction_traces_found_total` metric, not printed by the command. -Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for the query syntax and constraints. - -You can also bound a query-based redaction with `--start` and `--end` -(`now`, a relative offset such as `now-7d`, or an RFC3339 timestamp). -Both bounds are required, and they can't be combined with `--trace-id`. -A windowed redaction only scans blocks that overlap the range, -so a large tenant can be redacted in slices instead of holding compaction off for the whole run. -[[PR 7702](https://github.com/grafana/tempo/pull/7702)] +You can also bound a redaction to a time window with `--start` and `--end`, so a large tenant can be redacted in slices. [[PR 7702](https://github.com/grafana/tempo/pull/7702)] {{< admonition type="caution" >}} Don't submit a windowed redaction until every scheduler and worker is on Tempo 3.1. An older worker ignores the window and redacts every query match in the block, with no error. {{< /admonition >}} -### Backend scheduler redaction reliability +This release also includes several [backend scheduler redaction reliability](#backend-scheduler-redaction-reliability) fixes. -Query-based selection only helps if the redaction actually finishes. -This release closes several correctness gaps in the backend scheduler's redaction pipeline that could leave a job looking stuck, skip a compacted block, or pause compaction during a dry run: - -- A completed redaction batch now enters a short quiescence period before removal, so a block compacted in the window right after the last redaction job finishes still gets covered by a rescan instead of escaping redaction. [[PR 7695](https://github.com/grafana/tempo/pull/7695)] -- Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path where the body's `tenant_id` field was previously trusted. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] -- A dry-run redaction no longer disables the tenant's compaction and retention or arms a rescan; those now only happen for a redaction in apply mode, which actually rewrites blocks. [[PR 7700](https://github.com/grafana/tempo/pull/7700)] -- A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could leave a tenant's redaction looking perpetually in progress and block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] -- Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log make it observable when a redaction job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] -- A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode, so you can preview a redaction's blast radius before it runs, alongside a new Redaction row on the Backend Work dashboard. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] -- A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth (jobs enqueued and not yet assigned), so you can tell whether a redaction needs more workers. - Unlike `jobs_active`, which is capped by worker count, this is the signal that can drive scale-up. - Refer to [Key metrics](/docs/tempo//reference-tempo-architecture/components/compaction/#key-metrics). [[PR 7772](https://github.com/grafana/tempo/pull/7772)] - -The backend scheduler also got several scalability fixes so job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) - -Refer to [Compaction](/docs/tempo//reference-tempo-architecture/components/compaction/) for the redaction job lifecycle, metrics, and Backend Work dashboard. +Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for query syntax, constraints, and windowed redaction. ## TraceQL metrics extrapolation and arithmetic If you use head-based sampling to control costs, metrics derived from those traces reflect only the sampled fraction, not actual traffic. -Request rates, error counts, and throughput all undercount by the sampling factor. At 10% sampling, `{ } | rate()` reports 50 req/s when real traffic is 500 req/s. -Getting accurate numbers meant a parallel metrics pipeline, -the ingest-time metrics-generator, or multiplying by hand in every dashboard. -Tempo 3.1 lets TraceQL metrics extrapolate those counts from the W3C `tracestate` sampling probability on the read path. -The numbers come from traces you already stored, so you don't need extra series or a second pipeline. +Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, +so rates and counts reflect actual traffic without a separate metrics pipeline. Opt in per query with the experimental `with(extrapolate=true)` hint. -When a span carries an OpenTelemetry probability threshold in its tracestate, -it contributes `1 / sampling_probability` to `rate`, `count_over_time`, `sum_over_time`, `avg_over_time`, `histogram_over_time`, `quantile_over_time`, and `compare`. -That matches the metrics-generator's existing per-span multiplier behavior. -`min_over_time` and `max_over_time` are unaffected, and the hint requires vParquet4 or later. [[PR 7452](https://github.com/grafana/tempo/pull/7452)] -TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`). -You can combine aggregations in a single query instead of running two queries and stitching them client-side. -Dashboards stay simpler, and the time buckets line up. -Divide two rates to get an error rate, or scale a result by a constant. +TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`) +so you can combine aggregations in a single query instead of stitching results client-side. (PRs [#6866](https://github.com/grafana/tempo/pull/6866), [#7199](https://github.com/grafana/tempo/pull/7199), [#7409](https://github.com/grafana/tempo/pull/7409)) This query on head-sampled data returns an error rate that reflects real traffic: @@ -103,11 +64,7 @@ This query on head-sampled data returns an error rate that reflects real traffic ({status=error} | rate()) / ({} | rate()) with(extrapolate=true) ``` -Refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/) for the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the extrapolation hint. - -The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false`, or per-query with the hint `with(spanonly_fetch=false)`, if you hit a regression. Refer to [Faster read path](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path). [[PR 7179](https://github.com/grafana/tempo/pull/7179)] - -This release also fixes several TraceQL correctness issues: instant metrics queries now correctly reuse the per-block job results cache [[PR 7602](https://github.com/grafana/tempo/pull/7602)]; the vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `{ trace:rootService="..." } | rate()`, `span:childCount`, and array operations, or for metrics queries on event and link intrinsics [[PR 7508](https://github.com/grafana/tempo/pull/7508), [PR 7533](https://github.com/grafana/tempo/pull/7533)]; and `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. [[PR 7170](https://github.com/grafana/tempo/pull/7170)] +For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo//metrics-from-traces/metrics-queries/functions/#extrapolation). ## Metrics-generator and service graph improvements @@ -118,10 +75,11 @@ Service graphs get a new opt-in `traces_service_graph_connection_info` presence powered by new `service-graphs-*` subprocessors, for detecting service topology under heavy sampling, visualized in a new `tempo-service-graph.json` dashboard. [[PR 7202](https://github.com/grafana/tempo/pull/7202)] -An edge used to require both the client and server span from the same trace; -at low sample rates, one side is often missing and the connection never appears. -The new subprocessors detect service-to-service connections from individual spans, +Previously, an edge required both the client and server span from the same trace; +at low sample rates one side is often missing and the connection never appears. +The new subprocessors detect connections from individual spans, so the map stays more complete even when most traces are dropped. + Refer to [Connection information metric](/docs/tempo//metrics-from-traces/service_graphs/#connection-information-metric) to enable the subprocessor and query the gauge. Additional service graph improvements: @@ -135,7 +93,7 @@ At high ingest rates, that per-span cost is what makes the generator expensive t Metric names, labels, and values are unchanged. (PRs [#7584](https://github.com/grafana/tempo/pull/7584), [#7587](https://github.com/grafana/tempo/pull/7587)) As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior. -A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. Refer to [Metrics-generator](/docs/tempo//configuration/#metrics-generator). [[PR 7611](https://github.com/grafana/tempo/pull/7611)] +A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. [[PR 7611](https://github.com/grafana/tempo/pull/7611), [documentation](/docs/tempo//configuration/#metrics-generator)] A restart or scale-up no longer spends time processing spans that would never become metrics. Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing `tempo_ingest_group_partition_lag` series after a partition handoff. [[PR 7665](https://github.com/grafana/tempo/pull/7665)] That series is how you tell whether the generator is keeping up; @@ -175,10 +133,14 @@ Thanks to all three for this work. Refer to [Configure authentication and TLS](/docs/tempo//set-up-for-tracing/setup-tempo/configure-kafka/#configure-authentication-and-tls) for SASL and TLS, and [Ingest](/docs/tempo//configuration/#ingest) for `client_rack` and `producer_compression`. -The distributor also now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click "Forget" after a non-graceful Pod termination in most cases. Refer to [Tune the consistent hash rings](/docs/tempo//operations/manage-advanced-systems/consistent_hash_ring/#distributor). [[PR 7098](https://github.com/grafana/tempo/pull/7098)] - ## Trace diff and trace summary (experimental) +{{< admonition type="note" >}} +Trace diff and trace summary is an [experimental feature](https://grafana.com/docs/release-life-cycle/). +Engineering and on-call support is not available. +Experimental features might change or be removed in future releases and are not recommended for use in production environments. +{{< /admonition >}} + Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)] By default, the diff reports a compact summary alongside the full patch, up to 64 KiB; larger patches report that the patch was omitted rather than truncating it silently (`trace-summary-v0-composed`). [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] @@ -195,36 +157,71 @@ Refer to [Trace diff](/docs/tempo//api_docs/#trace-diff) for the The most important remaining features and enhancements in Tempo 3.1 are highlighted below. -- Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo//operations/verify-container-images/)) -- KEDA-based autoscaling is now available for live-store via a Prometheus trigger on expected bytes held, and for the metrics-generator in Jsonnet. - New top-level `autoscaling_prometheus_url` and `autoscaling_prometheus_tenant` fields configure the Prometheus source; set `autoscaling_prometheus_tenant` when the source is a multi-tenant system such as Grafana Mimir. - The live-store trigger uses `AverageValue`; using `Value` scales the cluster to `maxReplicas` regardless of load. - (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) -- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes on all query paths, a `tempo_querier_backend_processing_duration_seconds` histogram for time the querier spends processing backend blocks (excluding live-store data), and stats metrics on additional querier methods. +### TraceQL correctness + +- Instant metrics queries now correctly reuse the per-block job results cache. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] +- The vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `trace:rootService` and `span:childCount`, array operations, or metrics queries on event and link intrinsics. (PRs [#7508](https://github.com/grafana/tempo/pull/7508), [#7533](https://github.com/grafana/tempo/pull/7533)) +- `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. [[PR 7170](https://github.com/grafana/tempo/pull/7170)] + +### Performance improvements + +- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false` or per-query with `with(spanonly_fetch=false)` if you hit a regression. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path)] +- Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) +- The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) + +### Backend scheduler redaction reliability + +This release closes several correctness and scalability gaps in the backend scheduler's redaction pipeline: + +- A completed redaction batch now enters a short quiescence period before removal, so a block compacted right after the last job finishes still gets covered by a rescan. [[PR 7695](https://github.com/grafana/tempo/pull/7695)] +- Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] +- A dry-run redaction no longer disables the tenant's compaction and retention or arms a rescan; those now only happen for a redaction in apply mode. [[PR 7700](https://github.com/grafana/tempo/pull/7700)] +- A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] +- Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log surface when a job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] +- A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode. A new Redaction row on the Backend Work dashboard visualizes these. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] +- A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth. [[PR 7772](https://github.com/grafana/tempo/pull/7772), [documentation](/docs/tempo//reference-tempo-architecture/components/compaction/#key-metrics)] +- Job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) + +For the full redaction job lifecycle and metrics, refer to [Compaction](/docs/tempo//reference-tempo-architecture/components/compaction/). + +### Query and TraceQL + +- The trace-by-id v2 endpoint adds filtering support with `q` (TraceQL filter) and `keep_hierarchy` query parameters. + `match_depth` and `ancestor_depth` control how many descendant and ancestor hops to keep. + [[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo//api_docs/#query-v2)] +- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes, a `tempo_querier_backend_processing_duration_seconds` histogram, and stats metrics on additional querier methods. Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [#7571](https://github.com/grafana/tempo/pull/7571), [#7568](https://github.com/grafana/tempo/pull/7568), [documentation](/docs/tempo//reference-tempo-architecture/components/querier/#key-metrics)) -- The trace-by-id v2 endpoint adds a first pass of filtering support, with `q` (TraceQL filter) and `keep_hierarchy` query parameters to return only matching spans. - `match_depth` (default `0`) bounds how many descendant hops to keep, and `ancestor_depth` (default `-1`, only with `keep_hierarchy=true`) bounds ancestor hops. - `tempo-cli query trace-id` adds matching `--match-depth` and `--ancestor-depth` flags. - [[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo//api_docs/#query-v2)] -- The Tempo MCP server's documentation tools were expanded and refreshed: a new `docs-config` tool and `docs://config/overview` / `docs://config/reference` resources serve the configuration reference (generated from the default configuration, so it stays in sync with the code), and the TraceQL and metrics documentation served by the MCP server was updated to match current capabilities. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo//api_docs/mcp-server/#available-tools)) -- Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group (on a 2.4 GB block, a high-cardinality scan dropped from 67ms to 9ms); `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner, freeing space for active blocks. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) -- The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options for tuning connection behavior. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) -- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology from `traces_service_graph_connection_info`, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. - The Service Topology dashboard queries `traces_service_graph_request_total`. - (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo//operations/monitor/#dashboards)) -- A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] -- Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] -- An opt-in `engine_bytes_tracking` override reports protobuf-like span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`. - This isn't parquet bytes inspected, and it's disabled by default. +- An opt-in `engine_bytes_tracking` override reports span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`, so you can identify which queries scan the most data and plan capacity accordingly. [[PR 7689](https://github.com/grafana/tempo/pull/7689), [documentation](/docs/tempo//configuration/#overrides)] -- New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, so you can tune `max_input_blocks`. +- The Tempo MCP server adds a `docs-config` tool and refreshed TraceQL and metrics documentation resources. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo//api_docs/mcp-server/#available-tools)) + +### Configuration and build + +- New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, helping you spot oversized blocks and tune compaction settings. [[PR 7773](https://github.com/grafana/tempo/pull/7773)] +- A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] +- The distributor now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click "Forget" after a non-graceful Pod termination. [[PR 7098](https://github.com/grafana/tempo/pull/7098), [documentation](/docs/tempo//operations/manage-advanced-systems/consistent_hash_ring/#distributor)] +- Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance, so you can verify image authenticity before deploying. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo//operations/verify-container-images/)) +- KEDA-based autoscaling is now available for live-store and for the metrics-generator in Jsonnet. + Set `autoscaling_prometheus_tenant` when the Prometheus source is a multi-tenant system such as Grafana Mimir. + (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) +- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. + (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo//operations/monitor/#dashboards)) +- Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] ## Upgrade considerations When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps. +| Condition | Action | +| --- | --- | +| You write vParquet3 blocks | Change block version before upgrading. Refer to [vParquet3 deprecation enforced](#vparquet3-deprecation-enforced). | +| You use Redis cache | Review renamed YAML keys and removed Sentinel support. Refer to [Redis client rewrite](#redis-client-rewrite). | +| You depend on a fixed Trace-by-ID shard count | Set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Default value changes](#default-value-changes). | +| You depend on gRPC streaming responses larger than 1 MB | Set `max_grpc_streaming_packet_size` explicitly. Refer to [Default value changes](#default-value-changes). | +| You pin Docker image tags | Switch to immutable GAR tags. Refer to [Build and platform changes](#build-and-platform-changes). | + ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. @@ -238,46 +235,41 @@ storage: version: vParquet4 ``` -Refer to [Default block format is now vParquet5](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) and [Apache Parquet block format](/docs/tempo//configuration/parquet/). -[[PR 7775](https://github.com/grafana/tempo/pull/7775)] +Refer to [Default block format is now vParquet5](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) for upgrade steps and [Apache Parquet block format](/docs/tempo//configuration/parquet/) for format details. [[PR 7775](https://github.com/grafana/tempo/pull/7775)] ### vParquet3 deprecation enforced vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. Reads of existing vParquet3 blocks are unaffected. -[[PR 7858](https://github.com/grafana/tempo/pull/7858)] If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading. -Refer to [vParquet3 deprecation enforced](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) and [Apache Parquet block format](/docs/tempo//configuration/parquet/). - -### Redis client rewrite - -The experimental Redis cache client has been completely rewritten: Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. If you don't use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] - -### Query sharding: `blocks_per_shard` takes precedence over `query_shards` - -Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Trace by ID query sharding now scales with block count](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count). [[PR 7105](https://github.com/grafana/tempo/pull/7105)] - -### Metrics queries with identical start and end timestamps are now rejected - -As part of a fix to the metrics job results cache, a metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] - -### Default gRPC streaming packet size reduced +[[PR 7858](https://github.com/grafana/tempo/pull/7858)] -The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MB to 1 MB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. Refer to [Response larger than the max](/docs/tempo//troubleshooting/querying/response-too-large/). [[PR 7615](https://github.com/grafana/tempo/pull/7615)] +Refer to [vParquet3 deprecation enforced](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) for upgrade steps and [Apache Parquet block format](/docs/tempo//configuration/parquet/) for format details. -### Memcached idle connections stay open by default +### Redis client rewrite -Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. Refer to [Memcached cache connection defaults](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults). [[PR 7671](https://github.com/grafana/tempo/pull/7671)] +The experimental Redis cache client has been completely rewritten: +- Redis Cluster is now the default routing mode, +- Redis Sentinel support is removed, +- several YAML keys are renamed, and +- the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. + +If you don't use the Redis cache, no action is needed. +Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] -### Docker image tags +### Default value changes -Release artifacts no longer publish mutable Docker image tags; images are published only to immutable GAR repositories, and example configs are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] +- Query sharding: Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. [[PR 7105](https://github.com/grafana/tempo/pull/7105), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count)] +- Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] +- gRPC streaming packet size: The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MB to 1 MB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. [[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo//troubleshooting/querying/response-too-large/)] +- Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. [[PR 7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults)] -### Go version upgrade +### Build and platform changes -Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. +- Release artifacts no longer publish mutable Docker image tags; images are published only to immutable GAR repositories, and example configurations are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] +- Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. ## Security fixes @@ -288,20 +280,23 @@ Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). -- Bound per-trace slice preallocation during distributor rebatching. [[PR 7288](https://github.com/grafana/tempo/pull/7288)] -- Return a retryable status for transient errors writing to Kafka. [[PR 7506](https://github.com/grafana/tempo/pull/7506)] -- Cache empty tag-value results per block so a block with no matching values isn't re-scanned on every query. [[PR 7617](https://github.com/grafana/tempo/pull/7617)] +- Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [[PR 7787](https://github.com/grafana/tempo/pull/7787)] +- Fix classic histogram `_sum` series (for example, `traces_spanmetrics_latency_sum`) under-reporting via `increase()`/`rate()` for series with high churn. The `_sum` series was missing the zero-sample seed that `_count` and `_bucket` already had. [[PR 7811](https://github.com/grafana/tempo/pull/7811)] - Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. [[PR 7155](https://github.com/grafana/tempo/pull/7155)] -- Stop silently dropping a block's tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [[PR 7610](https://github.com/grafana/tempo/pull/7610)] -- Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn't page at `+Inf%` when no records have been produced yet. [[PR 7715](https://github.com/grafana/tempo/pull/7715)] - Fix a data race in the query frontend when dispatching request batches to queriers with `max_batch_size > 1`. [[PR 7664](https://github.com/grafana/tempo/pull/7664)] - Reject TraceQL queries larger than `max_query_expression_size_bytes` before parsing them. [[PR 7245](https://github.com/grafana/tempo/pull/7245), [documentation](/docs/tempo//configuration/#cap-the-maximum-query-length)] -- Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. [[PR 7271](https://github.com/grafana/tempo/pull/7271)] -- Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. [[PR 7662](https://github.com/grafana/tempo/pull/7662)] -- Fix a rare cache collision between instant and range metrics queries. [[PR 7290](https://github.com/grafana/tempo/pull/7290)] -- Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it's now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] - Fix `=nil` queries missing some values in TraceQL. [[PR 7345](https://github.com/grafana/tempo/pull/7345)] - Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). [[PR 7660](https://github.com/grafana/tempo/pull/7660)] - Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [[PR 7609](https://github.com/grafana/tempo/pull/7609)] -- Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [[PR 7787](https://github.com/grafana/tempo/pull/7787)] +- Fix a panic in TraceQL metrics-math queries (for example, `(A) / (B)`) when a sub-query's `by()` clause uses the maximum number of group-by attributes. [[PR 7831](https://github.com/grafana/tempo/pull/7831)] +- Cache empty tag-value results per block so a block with no matching values isn't re-scanned on every query. [[PR 7617](https://github.com/grafana/tempo/pull/7617)] +- Stop silently dropping a block's tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [[PR 7610](https://github.com/grafana/tempo/pull/7610)] +- Fix a rare cache collision between instant and range metrics queries. [[PR 7290](https://github.com/grafana/tempo/pull/7290)] +- Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. [[PR 7271](https://github.com/grafana/tempo/pull/7271)] - Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn't compacted or polled before it's complete. [[PR 7786](https://github.com/grafana/tempo/pull/7786)] +- Bound per-trace slice preallocation during distributor rebatching. [[PR 7288](https://github.com/grafana/tempo/pull/7288)] +- Return a retryable status for transient errors writing to Kafka. [[PR 7506](https://github.com/grafana/tempo/pull/7506)] +- Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn't page at `+Inf%` when no records have been produced yet. [[PR 7715](https://github.com/grafana/tempo/pull/7715)] +- Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. [[PR 7662](https://github.com/grafana/tempo/pull/7662)] +- Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it's now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] +- Fix the packaged configuration for deb/rpm installs. Also adds CI validation and an errors-only flag for `-config.verify`. [[PR 7830](https://github.com/grafana/tempo/pull/7830)] diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md index dc56ffbae40..af3703724d5 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md @@ -21,6 +21,7 @@ Tempo provides various command-line flags to configure its behavior when startin | `--config.file` | Configuration file to load | | | `--config.expand-env` | Whether to expand environment variables in config file | `false` | | `--config.verify` | Verify configuration and exit | `false` | +| `--config.verify-errors-only` | When used with `--config.verify`, exit successfully if the configuration has only warnings and no hard errors | `false` | ## Target flag @@ -156,6 +157,12 @@ Verify configuration without starting Tempo: tempo --config.file=/etc/tempo/config.yaml --config.verify ``` +Verify configuration but treat warnings as non-fatal (useful in CI pipelines): + +```bash +tempo --config.file=/etc/tempo/config.yaml --config.verify --config.verify-errors-only +``` + Print version information: ```bash From 60e3e686a6ca2c08a1519c5c7b762e4f4e33c566 Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Mon, 14 Sep 2026 10:28:46 -0400 Subject: [PATCH 7/8] Updates for Tiffany's comments --- docs/sources/tempo/release-notes/v3-1.md | 26 ++++++++++++------------ 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index 70ca1e56a98..45f3698de1d 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -22,7 +22,7 @@ This release gives you: - [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. - [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. - [Trace diff and trace summary (experimental)](#trace-diff-and-trace-summary-experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, `tempo-cli`, and the MCP server. -- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks still read, with no migration. +- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks remain readable, with no migration. These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). @@ -64,7 +64,7 @@ This query on head-sampled data returns an error rate that reflects real traffic ({status=error} | rate()) / ({} | rate()) with(extrapolate=true) ``` -For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo//metrics-from-traces/metrics-queries/functions/#extrapolation). +For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo//metrics-from-traces/metrics-queries/functions/#extrapolation-from-ingest-time-sampling-withextrapolatetrue-experimental). ## Metrics-generator and service graph improvements @@ -103,7 +103,7 @@ after a handoff it used to keep growing, which made health dashboards hard to tr The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. [[PR 7566](https://github.com/grafana/tempo/pull/7566)] -An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don't explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren't re-pruned. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override controls this independently of the cluster-wide default. [[PR 7693](https://github.com/grafana/tempo/pull/7693)] +An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don't explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren't re-pruned. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override lets individual tenants opt in or out of the default-on behavior, but only takes effect when the cluster-wide `span_pruning_enabled` is also `true`. [[PR 7693](https://github.com/grafana/tempo/pull/7693)] To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant `span_pruning_awareness` override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. [[PR 7532](https://github.com/grafana/tempo/pull/7532)] @@ -113,11 +113,11 @@ Refer to [Query V2](/docs/tempo//api_docs/#query-v2) for the requ In microservices mode, Tempo uses Kafka as a durable write-ahead log: traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. -Until 3.1, the Tempo Kafka client couldn't authenticate, encrypt in transit, prefer a local rack, or choose a compression codec. -If your cluster required SASL or TLS, you couldn't use this path at all. +Until 3.1, the Tempo Kafka client only supported SASL `PLAIN` authentication, couldn't encrypt in transit, prefer a local rack, or choose a compression codec. +If your cluster required TLS or a stronger SASL mechanism, you couldn't use this path at all. -Tempo 3.1 adds SASL authentication (`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, or `AWS_MSK_IAM`) and TLS/mutual-TLS, -so you can connect to authenticated, encrypted Kafka, including Amazon MSK. +Tempo 3.1 adds SASL `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, and `AWS_MSK_IAM` authentication alongside the existing `PLAIN` support, plus TLS and mutual-TLS, +so you can connect to encrypted Kafka clusters, including Amazon MSK. A `client_rack` option enables rack-aware fetching (KIP-392). Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, which cuts cross-zone data transfer cost on high-throughput pipelines. @@ -126,7 +126,7 @@ which Azure Event Hubs requires because it only supports `gzip`. (PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691)) Community contributors independently unblocked each of those gaps: -SASL and TLS from [@heytrav](https://github.com/heytrav), +additional SASL mechanisms and TLS from [@heytrav](https://github.com/heytrav), rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). Thanks to all three for this work. @@ -136,16 +136,16 @@ Refer to [Configure authentication and TLS](/docs/tempo//set-up-f ## Trace diff and trace summary (experimental) {{< admonition type="note" >}} -Trace diff and trace summary is an [experimental feature](https://grafana.com/docs/release-life-cycle/). +Trace diff and trace summary are [experimental features](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available. Experimental features might change or be removed in future releases and are not recommended for use in production environments. {{< /admonition >}} Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)] -By default, the diff reports a compact summary alongside the full patch, up to 64 KiB; larger patches report that the patch was omitted rather than truncating it silently (`trace-summary-v0-composed`). [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] +The HTTP API defaults to `trace-patch-v0` (the full span-level patch). A `trace-summary-v0-composed` format returns a compact summary alongside the patch, capping the patch at 64 KiB; larger patches report that the patch was omitted rather than truncating it silently. [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] -`tempo-cli` also gets trace diff support for local work: `trace diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] +`tempo-cli` also gets trace diff support for local work: `tempo-cli experimental trace-diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] The MCP server also exposes a `trace-diff` tool for complete-trace comparisons, with a compact composed summary by default and patches up to 64 KiB. @@ -263,12 +263,12 @@ Refer to [Redis cache configuration changes](/docs/tempo//set-up- - Query sharding: Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. [[PR 7105](https://github.com/grafana/tempo/pull/7105), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count)] - Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] -- gRPC streaming packet size: The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MB to 1 MB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. [[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo//troubleshooting/querying/response-too-large/)] +- gRPC streaming packet size: The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MiB to 1 MiB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. [[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo//troubleshooting/querying/response-too-large/)] - Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. [[PR 7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults)] ### Build and platform changes -- Release artifacts no longer publish mutable Docker image tags; images are published only to immutable GAR repositories, and example configurations are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] +- Release artifacts no longer publish mutable Docker image tags. Immutable GAR repositories are the primary release source, and images are also mirrored to Docker Hub. Example configurations are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] - Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. ## Security fixes From cf4e71c1ad95e6c7b23daa85b8d1e636821257cc Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Mon, 14 Sep 2026 12:40:47 -0400 Subject: [PATCH 8/8] docs: Address Copilot review comments on 3.1 release notes - Link CVEs to NVD advisories in the security section - Note that extrapolation requires vParquet4 or later blocks - Document unsafe_query_hints prerequisite for spanonly_fetch hint --- docs/sources/tempo/release-notes/v3-1.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md index 45f3698de1d..9d2bd4865d3 100644 --- a/docs/sources/tempo/release-notes/v3-1.md +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -52,6 +52,7 @@ At 10% sampling, `{ } | rate()` reports 50 req/s when real traffic is 500 req/s. Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, so rates and counts reflect actual traffic without a separate metrics pipeline. Opt in per query with the experimental `with(extrapolate=true)` hint. +Extrapolation requires vParquet4 or later blocks; queries against vParquet3 blocks ignore the hint. [[PR 7452](https://github.com/grafana/tempo/pull/7452)] TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`) @@ -126,7 +127,7 @@ which Azure Event Hubs requires because it only supports `gzip`. (PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691)) Community contributors independently unblocked each of those gaps: -additional SASL mechanisms and TLS from [@heytrav](https://github.com/heytrav), +SASL and TLS from [@heytrav](https://github.com/heytrav), rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). Thanks to all three for this work. @@ -165,7 +166,7 @@ The most important remaining features and enhancements in Tempo 3.1 are highligh ### Performance improvements -- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false` or per-query with `with(spanonly_fetch=false)` if you hit a regression. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path)] +- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with the `metrics_spanonly_fetch: false` override. You can also disable it per-query with `with(spanonly_fetch=false)`, but this hint requires [`unsafe_query_hints`](/docs/tempo//configuration/#overrides) to be enabled for the tenant. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path)] - Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) - The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) @@ -273,7 +274,7 @@ Refer to [Redis cache configuration changes](/docs/tempo//set-up- ## Security fixes -- Updated Go to 1.26.5 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to fix CVE-2026-39822, CVE-2026-42504, CVE-2026-27145, CVE-2026-42505, CVE-2026-42507, CVE-2026-46600, and CVE-2026-56852. [[PR 7641](https://github.com/grafana/tempo/pull/7641)] +- Updated Go to 1.26.5 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to address [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822), [CVE-2026-42504](https://nvd.nist.gov/vuln/detail/CVE-2026-42504), [CVE-2026-27145](https://nvd.nist.gov/vuln/detail/CVE-2026-27145), [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505), [CVE-2026-42507](https://nvd.nist.gov/vuln/detail/CVE-2026-42507), [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600), and [CVE-2026-56852](https://nvd.nist.gov/vuln/detail/CVE-2026-56852). [[PR 7641](https://github.com/grafana/tempo/pull/7641)] - Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] ## Bug fixes