docs: Draft release notes for 3.1 - #7806
Open
knylander-grafana wants to merge 8 commits into
Open
knylander-grafana wants to merge 8 commits into
knylander-grafana wants to merge 8 commits into
Conversation
knylander-grafana
requested review from
carles-grafana,
electron0zero,
ie-pham,
javiermolinar,
mapno,
mattdurham,
ruslan-mikhailov,
stoewer,
yvrhdn,
zalegrala and
zhxiaogg
as code owners
August 21, 2026 16:47
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the draft documentation page for Grafana Tempo 3.1 release notes under the docs release-notes section, summarizing major new capabilities, upgrade considerations, security fixes, and notable bug fixes for the upcoming release.
Changes:
- Adds a new
v3-1.mdrelease notes page with highlights (redaction-by-query, TraceQL metrics enhancements, metrics-generator/service-graph work, span pruning, Kafka ingestion improvements, and more). - Documents upgrade considerations and security fixes for 3.1, with links to deeper docs.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
docs/sources/tempo/release-notes/v3-1.md:45
- MEDIUM: This sentence reads as incomplete because it introduces additional details on the next line without punctuation. Adding a colon makes the lead-in clearer.
You can also bound a query-based redaction with `--start` and `--end`
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md:38
- HIGH — This section says existing vParquet3 blocks are still readable in 3.1, but the docs it points readers to for “version details” currently state that vParquet3 was removed in Tempo 3.0, which is a direct contradiction and will confuse upgrade planning.
Could we reconcile the Parquet-version docs (for example, docs/sources/tempo/configuration/parquet.md and docs/sources/tempo/reference-tempo-architecture/block-format.md) with the code’s current behavior (see tempodb/encoding/versioned.go), so readers don’t get mixed messages?
### Default block format is now vParquet5
Tempo 3.1 writes new blocks in vParquet5.
Existing vParquet4 and vParquet3 blocks still read, and no data migration is required.
[[PR 7775](https://github.com/grafana/tempo/pull/7775)]
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
knylander-grafana
force-pushed
the
3.1-draft-rel-notes
branch
from
September 9, 2026 01:49
410324f to
6c09bca
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The upgrade guide now contains redundant duplicate sections for trace-by-ID sharding, and a repeated grammatical issue appears in both docs files that should be corrected before publishing.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md:37
- "Existing vParquet4 and vParquet3 blocks still read" is ungrammatical and may be unclear to readers; rephrase to "are still readable".
Tempo 3.1 writes new blocks in vParquet5.
Existing vParquet4 and vParquet3 blocks still read, and no data migration is required.
docs/sources/tempo/release-notes/v3-1.md:231
- "Existing vParquet4 and vParquet3 blocks still read" is ungrammatical and may be unclear; rephrase to "are still readable".
Tempo 3.1 writes new blocks in vParquet5.
Existing vParquet4 and vParquet3 blocks still read, and no data migration is required.
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The Tempo 3.1 upgrade guide introduces a duplicated trace-by-ID sharding section (and related minor consistency issues) that should be resolved before publishing.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md:178
- This newly added Trace-by-ID sharding section duplicates the more detailed "Trace by ID query sharding now scales with block count" section later on the same page, including repeating the YAML example and defaults. Having both can confuse readers (two places to update) and makes the upgrade guide harder to maintain; consider keeping only the later detailed section.
### Trace by ID query sharding
Trace-by-ID lookups now shard dynamically based on the number of blocks in the blocklist rather than using a fixed shard count. A new `blocks_per_shard` option defaults to `30` and takes precedence over the deprecated `query_shards` setting. [[PR 7105](https://github.com/grafana/tempo/pull/7105)]
To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`:
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
Fix grammar, remove the duplicate query-sharding section, and point the release notes at the remaining upgrade heading.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The new 3.1 release-notes page contains many technical claims and defaults that warrant a more comprehensive human accuracy pass before publishing.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
docs/sources/tempo/release-notes/v3-1.md:268
- The config default for
max_grpc_streaming_packet_sizeis expressed in binary units (1,048,576 bytes = 1 MiB), and the linked troubleshooting page also describes it as 1 MiB. Using “MB” here is ambiguous; consider using “MiB” for consistency with the rest of the release notes (for example, “KiB” earlier in this page) and the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
javiermolinar
left a comment
javiermolinar
left a comment
Contributor
There was a problem hiding this comment.
There are too many low level details do we really want to add all of that?
Contributor
Author
|
@javiermolinar I've reworked a lot of the release notes based on your feedback. Here's the obligatory AI short summary of the major changes.
|
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved documentation corrections remain in the draft release notes.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (4)
docs/sources/tempo/release-notes/v3-1.md:148
- MEDIUM: Backticks make
trace difflook like the CLI command, but the documented command istempo-cli experimental trace-diff; copying this wording would lead users to invoke a command that does not exist. Could we use the exact command name here?
`tempo-cli` also gets trace diff support for local work: `trace diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)]
docs/sources/tempo/release-notes/v3-1.md:271
- MEDIUM: The claim that images are published only to immutable GAR repositories contradicts the release workflow and the verification page, which still publish/mirror the four images to Docker Hub. Could we describe the GAR tags as the immutable release source and mention the Docker Hub mirror instead of saying publication is GAR-only?
- Release artifacts no longer publish mutable Docker image tags; images are published only to immutable GAR repositories, and example configurations are pinned to Tempo 3.0.0. [[PR 7369](https://github.com/grafana/tempo/pull/7369)]
docs/sources/tempo/release-notes/v3-1.md:25
- The highlighted release-note bullet has a grammatical error: “existing vParquet4 blocks still read” is missing the adjective describing their readable state. Use “existing vParquet4 blocks remain readable” instead.
- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks still read, with no migration.
docs/sources/tempo/release-notes/v3-1.md:139
- Because “Trace diff and trace summary” is a compound subject, the sentence should use the plural “are” and “features.”
Trace diff and trace summary is an [experimental feature](https://grafana.com/docs/release-life-cycle/).
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Lite
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Outstanding release-note documentation corrections must be addressed before approval.
Review details
Suppressed comments (6)
docs/sources/tempo/release-notes/v3-1.md:146
- The HTTP API and
tempo-clidefault totrace-patch-v0, so this sentence incorrectly presentstrace-summary-v0-composedas the default for the whole feature. Restrict the composed-default statement to the MCP tool (described below), and state the interface-specific defaults so users do not expect a 64 KiB patch limit from the API or CLI.
By default, the diff reports a compact summary alongside the full patch, up to 64 KiB; larger patches report that the patch was omitted rather than truncating it silently (`trace-summary-v0-composed`). [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)]
docs/sources/tempo/release-notes/v3-1.md:168
- The per-query
with(spanonly_fetch=false)hint is an unsafe query hint;unsafe_query_hintsdefaults to false, so this opt-out is rejected unless that tenant override is enabled. Document that prerequisite, or direct users to the per-tenantmetrics_spanonly_fetch: falsesetting as the default-safe opt-out.
- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo/<TEMPO_VERSION>/release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false` or per-query with `with(spanonly_fetch=false)` if you hit a regression. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/#faster-read-path)]
docs/sources/tempo/release-notes/v3-1.md:25
- The phrase “existing vParquet4 blocks still read” is ungrammatical and reintroduces wording that the detailed upgrade guidance correctly avoids. Use “existing vParquet4 blocks remain readable.”
- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks still read, with no migration.
docs/sources/tempo/release-notes/v3-1.md:139
- The compound subject “Trace diff and trace summary” requires a plural verb. Change “is” to “are” in this experimental-feature notice.
Trace diff and trace summary is an [experimental feature](https://grafana.com/docs/release-life-cycle/).
docs/sources/tempo/release-notes/v3-1.md:106
- The per-tenant override is not independent of the cluster-wide setting: it only changes the default-on behavior when
span_pruning_enabledis enabled at the query frontend. As written, operators could expect this override to enable pruning on a cluster where pruning support is disabled; describe the gating relationship explicitly.
An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don't explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren't re-pruned. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override controls this independently of the cluster-wide default. [[PR 7693](https://github.com/grafana/tempo/pull/7693)]
docs/sources/tempo/release-notes/v3-1.md:276
- Security release notes must link each CVE to an advisory so operators can verify the affected vulnerability and remediation. This bullet lists seven CVEs as plain text, unlike the repository's linked-CVE format in the prior release notes; link each identifier to its NVD or GitHub advisory.
- Updated Go to 1.26.5 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to fix CVE-2026-39822, CVE-2026-42504, CVE-2026-27145, CVE-2026-42505, CVE-2026-42507, CVE-2026-46600, and CVE-2026-56852. [[PR 7641](https://github.com/grafana/tempo/pull/7641)]
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Three unresolved release-note documentation issues must be addressed.
Review details
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
docs/sources/tempo/release-notes/v3-1.md:276
- MEDIUM: The security section lists seven CVEs as plain text, so readers cannot verify the advisories from this release note. Existing security sections link each CVE to its NVD advisory (for example,
docs/sources/tempo/release-notes/version-2/v2-9.md:251); could we add those advisory links while retaining the PR link?
docs/sources/tempo/release-notes/v3-1.md:54
- MEDIUM: This describes
with(extrapolate=true)as generally available, but the corresponding changelog entry limits extrapolation to vParquet4 and later. Because Tempo 3.1 still reads vParquet3 blocks, users querying those legacy blocks could follow this guidance and get unsupported behavior; could we state the format restriction here?
Opt in per query with the experimental `with(extrapolate=true)` hint.
docs/sources/tempo/release-notes/v3-1.md:168
- MEDIUM: This per-query escape hatch is incomplete:
spanonly_fetchis an unsafe hint and requires the tenant'sunsafe_query_hintsoverride to be enabled (the metrics-query docs call this out atmetrics-queries/_index.md:122-133). Without that prerequisite, an operator following these release-note instructions may not actually disable the faster path. Could we mention the prerequisite here?
- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo/<TEMPO_VERSION>/release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with `metrics_spanonly_fetch: false` or per-query with `with(spanonly_fetch=false)` if you hit a regression. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/#faster-read-path)]
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
- Link CVEs to NVD advisories in the security section - Note that extrapolation requires vParquet4 or later blocks - Document unsafe_query_hints prerequisite for spanonly_fetch hint
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The configuration verification documentation needs correction, and two release-note nits remain.
Review details
Suppressed comments (3)
docs/sources/tempo/release-notes/v3-1.md:278
- PR #7153 fixes the vulnerability documented as GHSA-jw7f-3cgp-hgrr, but this security-fixes entry links only the PR. Add the advisory link so operators can assess whether they are affected, consistent with the release-notes security-linking requirement.
- Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [[PR 7153](https://github.com/grafana/tempo/pull/7153)]
docs/sources/tempo/release-notes/v3-1.md:117
- The verbs in this sentence are not parallel: "only supported ... authentication, couldn't encrypt ..., prefer ..., or choose ...". Use a semicolon before the
couldn'tclause so the three unsupported capabilities share that verb.
Until 3.1, the Tempo Kafka client only supported SASL `PLAIN` authentication, couldn't encrypt in transit, prefer a local rack, or choose a compression codec.
docs/sources/tempo/set-up-for-tracing/setup-tempo/command-line-flags.md:24
config.CheckConfig()also returns entries for invalid dedicated-column configuration, including one whose explanation says Tempo will not start, butconfigIsValidtreats every returned entry as a warning and returnserrorsOnlyfor the whole list. Consequently this command exits successfully for some configurations that cannot start, so the documentation's “only warnings and no hard errors” contract is not true until the flag distinguishes those cases (or the wording is changed).
| `--config.verify-errors-only` | When used with `--config.verify`, exit successfully if the configuration has only warnings and no hard errors | `false` |
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
|
||
| If you use head-based sampling to control costs, | ||
| metrics derived from those traces reflect only the sampled fraction, not actual traffic. | ||
| At 10% sampling, `{ } | rate()` reports 50 req/s when real traffic is 500 req/s. |
Contributor
There was a problem hiding this comment.
I think this is wrong, rate() is the rate of spans, no the rate of request/s. One request could emit more than one span. We could use smth like this instead
At 10% sampling, { } | rate() reports approximately 50 spans/s when the unsampled rate is 500 spans/s.
This query estimates the fraction of spans with error status, adjusted for sampling probability:
({status=error} | rate()) / ({} | rate()) with(extrapolate=true)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds draft release notes for upcoming Tempo 3.1 release.
Which issue(s) this PR fixes:
Fixes #7780
Checklist
.chloggen/(runmake chlog-new, ormake chlog-new FILENAME=<name>to override the default branch-name file; see.chloggen/README.md)