diff --git a/docs/sources/tempo/configuration/parquet.md b/docs/sources/tempo/configuration/parquet.md index 53da72211a6..d75701a02e1 100644 --- a/docs/sources/tempo/configuration/parquet.md +++ b/docs/sources/tempo/configuration/parquet.md @@ -12,7 +12,7 @@ This format is required for tags-based search as well as [TraceQL](../../traceql The columnar block format improves search performance and enables an ecosystem of tools, including [Tempo CLI](https://grafana.com/docs/tempo//operations/tempo_cli/#analyse-blocks), to access the underlying trace data. Starting in Tempo 3.1, Tempo writes new blocks in `vParquet5` by default. -Existing `vParquet4` and `vParquet3` blocks remain readable. +Existing `vParquet4` and `vParquet3` blocks remain readable, but `vParquet3` writes and compaction are no longer supported. No data migration is required. ## Considerations @@ -25,14 +25,15 @@ As soon as a block format version is enabled, Tempo starts writing data in that ## Block format versions -{{< admonition type="warning" >}} +Only Parquet-based formats are supported. + +### Removed and deprecated block formats + The `v2` block format has been removed in Tempo 3.0. -`vParquet3` is deprecated. + +`vParquet3` deprecation is enforced starting in Tempo 3.1: Tempo refuses to start if configured to write `vParquet3` blocks, and existing `vParquet3` blocks are no longer compacted. Tempo 3.x still reads existing `vParquet3` blocks. Write new blocks in `vParquet5` (default) or `vParquet4`. -{{< /admonition >}} - -Only Parquet-based formats are supported. ### vParquet5 diff --git a/docs/sources/tempo/release-notes/v3-1.md b/docs/sources/tempo/release-notes/v3-1.md new file mode 100644 index 00000000000..9d2bd4865d3 --- /dev/null +++ b/docs/sources/tempo/release-notes/v3-1.md @@ -0,0 +1,303 @@ +--- +title: Version 3.1 release notes +menuTitle: V3.1 +description: Release notes for Grafana Tempo 3.1 +weight: 650 +--- + +# Version 3.1 release notes + + + + + + +The Tempo team is pleased to announce the release of Grafana Tempo 3.1. + +This release gives you: + +- [Redaction by query](#redaction-by-query): Redact traces that match a TraceQL query instead of enumerating IDs, so a sensitive-data incident doesn't require hunting down every affected trace. +- [TraceQL metrics extrapolation and arithmetic](#traceql-metrics-extrapolation-and-arithmetic): Extrapolate counts from head-sampled traces so rates reflect actual traffic, and combine aggregations with arithmetic to calculate ratios like error rate in a single query. +- [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. +- [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option. +- [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. +- [Trace diff and trace summary (experimental)](#trace-diff-and-trace-summary-experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, `tempo-cli`, and the MCP server. +- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks remain readable, with no migration. + +These release notes highlight the most important features and bug fixes. +For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). + +## Redaction by query + +Tempo 3.0 introduced [trace redaction](/docs/tempo//release-notes/v3-0/#trace-redaction), but you had to supply an explicit list of trace IDs. +Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. [[PR 7663](https://github.com/grafana/tempo/pull/7663)] + +You can also bound a redaction to a time window with `--start` and `--end`, so a large tenant can be redacted in slices. [[PR 7702](https://github.com/grafana/tempo/pull/7702)] + +{{< admonition type="caution" >}} +Don't submit a windowed redaction until every scheduler and worker is on Tempo 3.1. +An older worker ignores the window and redacts every query match in the block, with no error. +{{< /admonition >}} + +This release also includes several [backend scheduler redaction reliability](#backend-scheduler-redaction-reliability) fixes. + +Refer to [Redact traces](/docs/tempo//operations/tempo_cli/#redact-traces) for query syntax, constraints, and windowed redaction. + +## TraceQL metrics extrapolation and arithmetic + +If you use head-based sampling to control costs, +metrics derived from those traces reflect only the sampled fraction, not actual traffic. +At 10% sampling, `{ } | rate()` reports 50 req/s when real traffic is 500 req/s. + +Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, +so rates and counts reflect actual traffic without a separate metrics pipeline. +Opt in per query with the experimental `with(extrapolate=true)` hint. +Extrapolation requires vParquet4 or later blocks; queries against vParquet3 blocks ignore the hint. +[[PR 7452](https://github.com/grafana/tempo/pull/7452)] + +TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`) +so you can combine aggregations in a single query instead of stitching results client-side. +(PRs [#6866](https://github.com/grafana/tempo/pull/6866), [#7199](https://github.com/grafana/tempo/pull/7199), [#7409](https://github.com/grafana/tempo/pull/7409)) + +This query on head-sampled data returns an error rate that reflects real traffic: + +```traceql +({status=error} | rate()) / ({} | rate()) with(extrapolate=true) +``` + +For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo//metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo//metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo//metrics-from-traces/metrics-queries/functions/#extrapolation-from-ingest-time-sampling-withextrapolatetrue-experimental). + +## Metrics-generator and service graph improvements + +If you use the metrics-generator, you get RED metrics and a service map from traces you already ingest. +The generator processes every span at ingest, so CPU, restart time, and dropped edges all affect the series you alert on. + +Service graphs get a new opt-in `traces_service_graph_connection_info` presence gauge, +powered by new `service-graphs-*` subprocessors, +for detecting service topology under heavy sampling, +visualized in a new `tempo-service-graph.json` dashboard. [[PR 7202](https://github.com/grafana/tempo/pull/7202)] +Previously, an edge required both the client and server span from the same trace; +at low sample rates one side is often missing and the connection never appears. +The new subprocessors detect connections from individual spans, +so the map stays more complete even when most traces are dropped. + +Refer to [Connection information metric](/docs/tempo//metrics-from-traces/service_graphs/#connection-information-metric) to enable the subprocessor and query the gauge. + +Additional service graph improvements: + +- Service graphs now recognize the `db.system.name` attribute (the OpenTelemetry v1.30.0 rename of `db.system`) for database node detection and virtual node naming. + Both attributes remain supported, so database nodes still appear after you upgrade to OpenTelemetry SDKs that emit the renamed attribute. [[PR 7697](https://github.com/grafana/tempo/pull/7697), [documentation](/docs/tempo//metrics-from-traces/service_graphs/#database-name-attributes)] +- Expired service-graph edges are now labeled with the unmatched span kind, so you can tell which side of a connection was missing instead of an edge disappearing with no explanation. [[PR 7709](https://github.com/grafana/tempo/pull/7709), [documentation](/docs/tempo//troubleshooting/metrics-generator/#expired-edges)] + +The span-metrics and service-graphs processors were both reworked to build series labels through the metrics-generator registry's pooled, borrowed-label path, cutting per-span and per-edge CPU and allocations. +At high ingest rates, that per-span cost is what makes the generator expensive to run. +Metric names, labels, and values are unchanged. (PRs [#7584](https://github.com/grafana/tempo/pull/7584), [#7587](https://github.com/grafana/tempo/pull/7587)) +As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior. + +A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. [[PR 7611](https://github.com/grafana/tempo/pull/7611), [documentation](/docs/tempo//configuration/#metrics-generator)] +A restart or scale-up no longer spends time processing spans that would never become metrics. +Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing `tempo_ingest_group_partition_lag` series after a partition handoff. [[PR 7665](https://github.com/grafana/tempo/pull/7665)] +That series is how you tell whether the generator is keeping up; +after a handoff it used to keep growing, which made health dashboards hard to trust. + +## Span pruning for trace-by-id v2 + +The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. [[PR 7566](https://github.com/grafana/tempo/pull/7566)] + +An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don't explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren't re-pruned. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override lets individual tenants opt in or out of the default-on behavior, but only takes effect when the cluster-wide `span_pruning_enabled` is also `true`. [[PR 7693](https://github.com/grafana/tempo/pull/7693)] + +To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant `span_pruning_awareness` override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. [[PR 7532](https://github.com/grafana/tempo/pull/7532)] + +Refer to [Query V2](/docs/tempo//api_docs/#query-v2) for the request parameters and [Query-frontend](/docs/tempo//configuration/#query-frontend) for the cluster-wide and per-tenant settings. + +## Kafka ingestion improvements + +In microservices mode, Tempo uses Kafka as a durable write-ahead log: +traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. +Until 3.1, the Tempo Kafka client only supported SASL `PLAIN` authentication, couldn't encrypt in transit, prefer a local rack, or choose a compression codec. +If your cluster required TLS or a stronger SASL mechanism, you couldn't use this path at all. + +Tempo 3.1 adds SASL `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, and `AWS_MSK_IAM` authentication alongside the existing `PLAIN` support, plus TLS and mutual-TLS, +so you can connect to encrypted Kafka clusters, including Amazon MSK. +A `client_rack` option enables rack-aware fetching (KIP-392). +Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, +which cuts cross-zone data transfer cost on high-throughput pipelines. +An `ingest.kafka.producer_compression` option overrides the producer's compression codec, +which Azure Event Hubs requires because it only supports `gzip`. +(PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691)) + +Community contributors independently unblocked each of those gaps: +SASL and TLS from [@heytrav](https://github.com/heytrav), +rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), +and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). +Thanks to all three for this work. + +Refer to [Configure authentication and TLS](/docs/tempo//set-up-for-tracing/setup-tempo/configure-kafka/#configure-authentication-and-tls) for SASL and TLS, and [Ingest](/docs/tempo//configuration/#ingest) for `client_rack` and `producer_compression`. + +## Trace diff and trace summary (experimental) + +{{< admonition type="note" >}} +Trace diff and trace summary are [experimental features](https://grafana.com/docs/release-life-cycle/). +Engineering and on-call support is not available. +Experimental features might change or be removed in future releases and are not recommended for use in production environments. +{{< /admonition >}} + +Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)] + +The HTTP API defaults to `trace-patch-v0` (the full span-level patch). A `trace-summary-v0-composed` format returns a compact summary alongside the patch, capping the patch at 64 KiB; larger patches report that the patch was omitted rather than truncating it silently. [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] + +`tempo-cli` also gets trace diff support for local work: `tempo-cli experimental trace-diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] + +The MCP server also exposes a `trace-diff` tool for complete-trace comparisons, +with a compact composed summary by default and patches up to 64 KiB. +[[PR 7785](https://github.com/grafana/tempo/pull/7785)] + +Refer to [Trace diff](/docs/tempo//api_docs/#trace-diff) for the HTTP API, [Compare traces](/docs/tempo//api_docs/mcp-server/#compare-traces) for the MCP tool, and [Experimental trace diff](/docs/tempo//operations/tempo_cli/#experimental-trace-diff) for the CLI. + +## Features and enhancements + +The most important remaining features and enhancements in Tempo 3.1 are highlighted below. + +### TraceQL correctness + +- Instant metrics queries now correctly reuse the per-block job results cache. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] +- The vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `trace:rootService` and `span:childCount`, array operations, or metrics queries on event and link intrinsics. (PRs [#7508](https://github.com/grafana/tempo/pull/7508), [#7533](https://github.com/grafana/tempo/pull/7533)) +- `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. [[PR 7170](https://github.com/grafana/tempo/pull/7170)] + +### Performance improvements + +- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo//release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with the `metrics_spanonly_fetch: false` override. You can also disable it per-query with `with(spanonly_fetch=false)`, but this hint requires [`unsafe_query_hints`](/docs/tempo//configuration/#overrides) to be enabled for the tenant. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo//metrics-from-traces/metrics-queries/#faster-read-path)] +- Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) +- The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//configuration/#cache)) + +### Backend scheduler redaction reliability + +This release closes several correctness and scalability gaps in the backend scheduler's redaction pipeline: + +- A completed redaction batch now enters a short quiescence period before removal, so a block compacted right after the last job finishes still gets covered by a rescan. [[PR 7695](https://github.com/grafana/tempo/pull/7695)] +- Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] +- A dry-run redaction no longer disables the tenant's compaction and retention or arms a rescan; those now only happen for a redaction in apply mode. [[PR 7700](https://github.com/grafana/tempo/pull/7700)] +- A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] +- Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log surface when a job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] +- A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode. A new Redaction row on the Backend Work dashboard visualizes these. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] +- A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth. [[PR 7772](https://github.com/grafana/tempo/pull/7772), [documentation](/docs/tempo//reference-tempo-architecture/components/compaction/#key-metrics)] +- Job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) + +For the full redaction job lifecycle and metrics, refer to [Compaction](/docs/tempo//reference-tempo-architecture/components/compaction/). + +### Query and TraceQL + +- The trace-by-id v2 endpoint adds filtering support with `q` (TraceQL filter) and `keep_hierarchy` query parameters. + `match_depth` and `ancestor_depth` control how many descendant and ancestor hops to keep. + [[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo//api_docs/#query-v2)] +- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes, a `tempo_querier_backend_processing_duration_seconds` histogram, and stats metrics on additional querier methods. + Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). + (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [#7571](https://github.com/grafana/tempo/pull/7571), [#7568](https://github.com/grafana/tempo/pull/7568), [documentation](/docs/tempo//reference-tempo-architecture/components/querier/#key-metrics)) +- An opt-in `engine_bytes_tracking` override reports span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`, so you can identify which queries scan the most data and plan capacity accordingly. + [[PR 7689](https://github.com/grafana/tempo/pull/7689), [documentation](/docs/tempo//configuration/#overrides)] +- The Tempo MCP server adds a `docs-config` tool and refreshed TraceQL and metrics documentation resources. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo//api_docs/mcp-server/#available-tools)) + +### Configuration and build + +- New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, helping you spot oversized blocks and tune compaction settings. + [[PR 7773](https://github.com/grafana/tempo/pull/7773)] +- A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] +- The distributor now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click "Forget" after a non-graceful Pod termination. [[PR 7098](https://github.com/grafana/tempo/pull/7098), [documentation](/docs/tempo//operations/manage-advanced-systems/consistent_hash_ring/#distributor)] +- Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance, so you can verify image authenticity before deploying. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo//operations/verify-container-images/)) +- KEDA-based autoscaling is now available for live-store and for the metrics-generator in Jsonnet. + Set `autoscaling_prometheus_tenant` when the Prometheus source is a multi-tenant system such as Grafana Mimir. + (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) +- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. + (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo//operations/monitor/#dashboards)) +- Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] + +## Upgrade considerations + +When [upgrading](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps. + +| Condition | Action | +| --- | --- | +| You write vParquet3 blocks | Change block version before upgrading. Refer to [vParquet3 deprecation enforced](#vparquet3-deprecation-enforced). | +| You use Redis cache | Review renamed YAML keys and removed Sentinel support. Refer to [Redis client rewrite](#redis-client-rewrite). | +| You depend on a fixed Trace-by-ID shard count | Set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Default value changes](#default-value-changes). | +| You depend on gRPC streaming responses larger than 1 MB | Set `max_grpc_streaming_packet_size` explicitly. Refer to [Default value changes](#default-value-changes). | +| You pin Docker image tags | Switch to immutable GAR tags. Refer to [Build and platform changes](#build-and-platform-changes). | + +### Default block format is now vParquet5 + +Tempo 3.1 writes new blocks in vParquet5. +Existing vParquet4 blocks are still readable, and no data migration is required. +To keep writing vParquet4, set the block version explicitly: + +```yaml +storage: + trace: + block: + version: vParquet4 +``` + +Refer to [Default block format is now vParquet5](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) for upgrade steps and [Apache Parquet block format](/docs/tempo//configuration/parquet/) for format details. [[PR 7775](https://github.com/grafana/tempo/pull/7775)] + +### vParquet3 deprecation enforced + +vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. +Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. +Reads of existing vParquet3 blocks are unaffected. + +If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading. +[[PR 7858](https://github.com/grafana/tempo/pull/7858)] + +Refer to [vParquet3 deprecation enforced](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) for upgrade steps and [Apache Parquet block format](/docs/tempo//configuration/parquet/) for format details. + +### Redis client rewrite + +The experimental Redis cache client has been completely rewritten: +- Redis Cluster is now the default routing mode, +- Redis Sentinel support is removed, +- several YAML keys are renamed, and +- the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. + +If you don't use the Redis cache, no action is needed. +Refer to [Redis cache configuration changes](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] + +### Default value changes + +- Query sharding: Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. [[PR 7105](https://github.com/grafana/tempo/pull/7105), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count)] +- Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] +- gRPC streaming packet size: The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MiB to 1 MiB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. [[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo//troubleshooting/querying/response-too-large/)] +- Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. [[PR 7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo//set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults)] + +### Build and platform changes + +- Release artifacts no longer publish mutable Docker image tags. Immutable GAR repositories are the primary release source, and images are also mirrored to Docker Hub. Example configurations are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)] +- Tempo 3.1 upgrades to Go 1.26.5. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. + +## Security fixes + +- Updated Go to 1.26.5 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to address [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822), [CVE-2026-42504](https://nvd.nist.gov/vuln/detail/CVE-2026-42504), [CVE-2026-27145](https://nvd.nist.gov/vuln/detail/CVE-2026-27145), [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505), [CVE-2026-42507](https://nvd.nist.gov/vuln/detail/CVE-2026-42507), [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600), and [CVE-2026-56852](https://nvd.nist.gov/vuln/detail/CVE-2026-56852). [[PR 7641](https://github.com/grafana/tempo/pull/7641)] +- Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] + +## Bug fixes + +For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). + +- Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [[PR 7787](https://github.com/grafana/tempo/pull/7787)] +- Fix classic histogram `_sum` series (for example, `traces_spanmetrics_latency_sum`) under-reporting via `increase()`/`rate()` for series with high churn. The `_sum` series was missing the zero-sample seed that `_count` and `_bucket` already had. [[PR 7811](https://github.com/grafana/tempo/pull/7811)] +- Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. [[PR 7155](https://github.com/grafana/tempo/pull/7155)] +- Fix a data race in the query frontend when dispatching request batches to queriers with `max_batch_size > 1`. [[PR 7664](https://github.com/grafana/tempo/pull/7664)] +- Reject TraceQL queries larger than `max_query_expression_size_bytes` before parsing them. [[PR 7245](https://github.com/grafana/tempo/pull/7245), [documentation](/docs/tempo//configuration/#cap-the-maximum-query-length)] +- Fix `=nil` queries missing some values in TraceQL. [[PR 7345](https://github.com/grafana/tempo/pull/7345)] +- Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). [[PR 7660](https://github.com/grafana/tempo/pull/7660)] +- Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [[PR 7609](https://github.com/grafana/tempo/pull/7609)] +- Fix a panic in TraceQL metrics-math queries (for example, `(A) / (B)`) when a sub-query's `by()` clause uses the maximum number of group-by attributes. [[PR 7831](https://github.com/grafana/tempo/pull/7831)] +- Cache empty tag-value results per block so a block with no matching values isn't re-scanned on every query. [[PR 7617](https://github.com/grafana/tempo/pull/7617)] +- Stop silently dropping a block's tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [[PR 7610](https://github.com/grafana/tempo/pull/7610)] +- Fix a rare cache collision between instant and range metrics queries. [[PR 7290](https://github.com/grafana/tempo/pull/7290)] +- Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. [[PR 7271](https://github.com/grafana/tempo/pull/7271)] +- Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn't compacted or polled before it's complete. [[PR 7786](https://github.com/grafana/tempo/pull/7786)] +- Bound per-trace slice preallocation during distributor rebatching. [[PR 7288](https://github.com/grafana/tempo/pull/7288)] +- Return a retryable status for transient errors writing to Kafka. [[PR 7506](https://github.com/grafana/tempo/pull/7506)] +- Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn't page at `+Inf%` when no records have been produced yet. [[PR 7715](https://github.com/grafana/tempo/pull/7715)] +- Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. [[PR 7662](https://github.com/grafana/tempo/pull/7662)] +- Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it's now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] +- Fix the packaged configuration for deb/rpm installs. Also adds CI validation and an errors-only flag for `-config.verify`. [[PR 7830](https://github.com/grafana/tempo/pull/7830)] diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md index dc56ffbae40..af3703724d5 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md @@ -21,6 +21,7 @@ Tempo provides various command-line flags to configure its behavior when startin | `--config.file` | Configuration file to load | | | `--config.expand-env` | Whether to expand environment variables in config file | `false` | | `--config.verify` | Verify configuration and exit | `false` | +| `--config.verify-errors-only` | When used with `--config.verify`, exit successfully if the configuration has only warnings and no hard errors | `false` | ## Target flag @@ -156,6 +157,12 @@ Verify configuration without starting Tempo: tempo --config.file=/etc/tempo/config.yaml --config.verify ``` +Verify configuration but treat warnings as non-fatal (useful in CI pipelines): + +```bash +tempo --config.file=/etc/tempo/config.yaml --config.verify --config.verify-errors-only +``` + Print version information: ```bash diff --git a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md index c92f87eb8ce..57279023b58 100644 --- a/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md +++ b/docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md @@ -31,11 +31,48 @@ You can check your configuration options using the [`status` API endpoint](https ## Upgrade to Tempo 3.1 +### Default block format is now vParquet5 + +Tempo 3.1 writes new blocks in vParquet5. +Existing vParquet4 blocks are still readable, and no data migration is required. +[[PR 7775](https://github.com/grafana/tempo/pull/7775)] + +To keep writing vParquet4, set the block version explicitly: + +```yaml +storage: + trace: + block: + version: vParquet4 +``` + +Refer to [Apache Parquet block format](/docs/tempo//configuration/parquet/) for version details. + +### vParquet3 deprecation enforced + +vParquet3 deprecation is now enforced in Tempo 3.1 [[PR 7858](https://github.com/grafana/tempo/pull/7858)]. +This means that: + +- Tempo refuses to start if configured to write vParquet3 blocks. +- Existing vParquet3 blocks are no longer compacted. +- Reads of existing vParquet3 blocks are unaffected. + +If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading: + +```yaml +storage: + trace: + block: + version: vParquet5 +``` + +Refer to [Apache Parquet block format](/docs/tempo//configuration/parquet/) for version details. + ### Redis cache configuration changes Tempo upgrades the Redis cache client to [`github.com/redis/go-redis/v9`](https://github.com/redis/go-redis) and reworks the cache configuration. Redis Cluster is now the default routing mode, Redis Sentinel support is removed, several YAML keys are renamed, and the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. [[PR 7337](https://github.com/grafana/tempo/pull/7337)] -If you do not use the Redis cache, no action is needed. +If you don't use the Redis cache, no action is needed. #### Opt single-node Redis into the single-node client