Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,14 @@ much longer to build and is what makes the results worth citing.

## Status

Not yet published. Six protocols, twelve backends and eighty-one checks run
unattended from one command; the write-up that has to precede publication is
not done.
Not yet published, and the reason is now a date rather than a task. Six
protocols, twelve backends and eighty checks run unattended from one command.
Every divergence is filed upstream or has a recorded reason for not being, and
every check that records a behaviour without judging it carries an open
adjudication issue. What is left is the one thing work cannot finish early: the
matrix is republished quarterly, and a claim to be maintained quarterly needs a
second quarter to have happened. `v1.0.0` waits for it; `v0.9.0` is the corpus
as it stands.

Logs came first: two protocols, six stores, twenty-four checks. Metrics
followed, and they are where silent alteration does the most damage — a wrong
Expand All @@ -110,9 +115,12 @@ case that found it, with the version and the exact request.

Each is filed with the project it concerns, with a reproduction that needs
nothing but `curl` — `specmatrix encode` produces the body for the protocols
that exist on the wire only as protobuf — and each is linked from the check
that found it. `CONTRIBUTING.md` requires that a maintainer learns about a
finding from their own tracker rather than from a comparison table.
that exist on the wire only as protobuf — and the issue URL goes in the
check's `rule.observed` list, so a reader can see the bug the check caught.
`CONTRIBUTING.md` requires that a maintainer learns about a finding from their
own tracker rather than from a comparison table. Fifteen of the twenty-three
divergences are filed and linked; the eight outstanding are all in the
`loki-push` suite, the newest one, and are why no page is published yet.

Loki's push API joined the same three log stores that already answer OTLP —
Loki itself, VictoriaLogs, OpenObserve — read back through the same
Expand Down Expand Up @@ -189,7 +197,7 @@ store you are willing to have written to, and read
## Running it in CI, without cloning this repository

```yaml
- uses: DeviousCardi/specmatrix@v1.0.0
- uses: DeviousCardi/specmatrix@v0.9.0
with:
backend: loki # or path/to/your-adapter.yaml for one not carried here
suite: otlp-logs
Expand Down
7 changes: 6 additions & 1 deletion backends/greptimedb.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@
# Confirmed by hand against greptime/greptimedb:v1.2.0 on 2026-09-09:
#
# $ curl -s localhost:4000/status
# {"source_time":"2025-09-28T08:21:06Z","commit":"4bb9ceb...","version":"0.17.2",...}
# {"commit":"29b743ab95568a47884ef0e8c42d579f0b0222e9","branch":"",
# "rustc_version":"rustc 1.96.0-nightly (ac7f9ec7d 2026-03-20)",
# "hostname":"d7cd5e0e68af","version":"1.2.0"}
#
# `/status` no longer carries `source_time`; the field was present on 0.17.2
# and is gone on 1.2.0. The version pointer is `/version`, which is unchanged.
#
# GreptimeDB claims the Prometheus remote-write and query APIs under a
# /v1/prometheus prefix. It stores metrics in its own table model rather than a
Expand Down
25 changes: 24 additions & 1 deletion backends/jaeger.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,33 @@
# `exact`, for this reason and this reason only.

name: jaeger
# Jaeger 2.x is an OpenTelemetry Collector distribution, and it has no version
# endpoint on either the OTLP port or the query port: `/api/version` and
# `/api/v3/version` both 404, and `/version` is swallowed by the UI's
# single-page-app catch-all and answers HTML. Without a version this column
# published as a blank, which is the one thing `docs/BACKENDS.md` says a column
# may not do — a matrix without versions is a claim about the past that reads
# as a claim about the present.
#
# The version is on the Collector's internal telemetry endpoint instead, which
# is why 8888 is published alongside the query port. Confirmed by hand against
# jaegertracing/jaeger:2.20.0 on 2026-09-09:
#
# $ curl -s localhost:8888/metrics | grep '^target_info'
# target_info{service_instance_id="8e20555c-5551-4cf7-a35d-551ff5eabf95",
# service_name="jaeger",service_version="v2.20.0"} 1
#
# Read with a pattern rather than a field, the same way VictoriaLogs is, since
# the response is Prometheus text and not JSON. The captured value keeps the
# `v` prefix the store itself reports.
version_from:
request: GET http://localhost:8888/metrics
pattern: 'target_info\{[^}]*service_version="([^"]+)"'

container:
image: jaegertracing/jaeger:2.20.0
port: 4318
extra_ports: [16686]
extra_ports: [16686, 8888]
env:
COLLECTOR_OTLP_ENABLED: "true"
ready:
Expand Down
3 changes: 2 additions & 1 deletion backends/mimir.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
# Confirmed by hand against grafana/mimir:3.2.0 on 2026-09-09:
#
# $ curl -s localhost:9009/prometheus/api/v1/status/buildinfo
# {"status":"success","data":{"application":"Grafana Mimir","version":"2.17.1",...}}
# {"status":"success","data":{"application":"Grafana Mimir","version":"3.2.0",
# "revision":"9ab70ccf",...}}
#
# Mimir writes at /api/v1/push, not /api/v1/write, and answers PromQL under a
# /prometheus prefix. Both are renames of the same protocol and belong here
Expand Down
3 changes: 2 additions & 1 deletion backends/prometheus.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
# Confirmed by hand against prom/prometheus:v3.14.0 on 2026-09-09:
#
# $ curl -s localhost:9090/api/v1/status/buildinfo
# {"status":"success","data":{"version":"3.6.0",...}}
# {"status":"success","data":{"version":"3.14.0",
# "revision":"d7598b7141418fa35be2b5ec5d0fefb634199610",...}}
#
# $ curl -sG localhost:9090/api/v1/query \
# --data-urlencode 'query=specmatrix_gauge{specmatrix_run="sm-..."}'
Expand Down
4 changes: 4 additions & 0 deletions cases/loki-push/duplicate-line-same-timestamp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,7 @@ notes: >
Confirmed against grafana/loki:3.7.7 on 2026-09-09: both lines are accepted
and both are queryable. `present` because this records which of the two a
store keeps, for a store that only keeps one; Loki itself keeps both.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/14, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
12 changes: 12 additions & 0 deletions cases/loki-push/empty-push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,15 @@ notes: >
rather than treated as the OTLP/remote-write pattern of "empty is ordinary
traffic": Loki's own reference behaviour is to refuse it, so a receiver
that refuses it too is agreeing with the reference, not failing a rule.

VictoriaLogs v1.52.0: ALTER. Accepted with 204 where Loki refuses with 422.
Confirmed by hand on 2026-09-09.

Deliberately not filed upstream. Nothing is lost and nothing is altered — the
push carried no records to lose — so the divergence is that VictoriaLogs is
more permissive than the reference, not that it mishandled data. Accepting an
empty push is also what OTLP requires of its own receivers ("if the server
receives an empty request ... the server SHOULD respond with success"), so a
report asking VictoriaLogs to start refusing it would be asking for stricter
behaviour with no benefit to a sender. Recorded here so the row on the page
can be read without anyone having to re-derive why no issue is linked.
14 changes: 14 additions & 0 deletions cases/loki-push/label-name-invalid.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,17 @@ notes: >
label is kept as sent — `1zone` appears as a top-level column on the stored
record. A sender relying on refusal to catch a malformed label gets no
signal from OpenObserve that Loki would have given it.

VictoriaLogs v1.52.0: ALTER, same as OpenObserve — accepted with 204 where
Loki refuses with 400. Confirmed by hand on 2026-09-09, and the label is kept
intact rather than dropped or renamed: the stored record carries
`"1zone":"a"` as a top-level field and `_stream` reads
`{1zone="a",service_name="vlrepro",specmatrix_run="repro-label"}`.

Deliberately not filed upstream, for either store. The label name is invalid
under Prometheus's grammar, which is where Loki's rule comes from, but both
stores keep the data they were sent and both let it be queried back. What a
sender loses is a validation signal it would have got from Loki, not a label.
That is worth recording on the page and is too thin to file as a defect
against two projects; a store is entitled to a looser label grammar than the
reference as long as it says so, and neither claims strict Loki parity.
4 changes: 4 additions & 0 deletions cases/loki-push/label-value-empty.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,7 @@ notes: >
`zone` label is dropped from the stream entirely — `{detected_level=...,
service_name=..., specmatrix_run=...}`, no `zone` key at all, matching the
Prometheus convention even though nothing documents that it must.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/15, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
1 change: 1 addition & 0 deletions cases/loki-push/line-invalid-utf8.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ rule:
replacement. Accepting and truncating the line is an ALTER regardless.
observed:
- https://github.com/grafana/loki/issues/24423
- https://github.com/VictoriaMetrics/VictoriaLogs/issues/1780

send:
format: loki-json
Expand Down
4 changes: 4 additions & 0 deletions cases/loki-push/out-of-order-within-stream.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,7 @@ notes: >
configuration: accepted (204), and both lines are queryable. Recent Loki
defaults to accepting a modest amount of out-of-order writes rather than
requiring -ingester.unordered-writes to be turned on for one this small.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/16, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/loki-push/structured-metadata-roundtrip.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,7 @@ notes: >
the value survives and is findable, and it is: `{... trace_id="deadbeef..."}`.
A store that kept it queryable only via a metadata-specific LogQL filter
would still pass; one that dropped it would not.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/17, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
2 changes: 2 additions & 0 deletions cases/loki-push/timestamp-nanosecond-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ rule:
epoch, so nanosecond precision is the field's native resolution rather
than a special case a store has to preserve deliberately. What is checked
is that a store claiming this API keeps the digits it was given.
observed:
- https://github.com/openobserve/openobserve/issues/14344

send:
format: loki-json
Expand Down
7 changes: 7 additions & 0 deletions cases/loki-push/timestamp-older-than-window.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,13 @@ rule:
not keep has a conformant answer available — refuse it — and this is the
de-facto reference for what that answer looks like when it names the
reason.
observed:
# OpenObserve drops the line silently on this endpoint while reporting the
# same drop on its own OTLP endpoint.
- https://github.com/openobserve/openobserve/issues/14345
# VictoriaLogs does the same; the push-API reproduction is a comment on the
# existing issue for the OTLP path, since it is one defect on two endpoints.
- https://github.com/VictoriaMetrics/VictoriaLogs/issues/1777

send:
format: loki-json
Expand Down
4 changes: 4 additions & 0 deletions cases/otlp-logs/body-empty-string.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,7 @@ notes: >

All five stores return the empty string they were given, confirmed
2026-09-08. The answer turned out to be unanimous.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/18, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
6 changes: 5 additions & 1 deletion cases/otlp-logs/body-kvlist.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ rule:
the structure will not find it.

observed:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11525
- https://github.com/VictoriaMetrics/VictoriaLogs/issues/1778

send:
format: otlp-json
Expand Down Expand Up @@ -51,3 +51,7 @@ notes: >
does not refuse the record: it stores a diagnostic string about its own data
model in the field where the log message belongs. A dashboard renders that as
the log line, and nothing errored at ingest.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/19, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
34 changes: 34 additions & 0 deletions cases/otlp-logs/empty-batch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,37 @@ notes: >

Loki 3.1.1: PASS, 204. Four of five stores accept an empty export; Quickwit
0.8.2 alone answers 500.

Loki 3.7.7: REJECT, 422 — a change from 3.1.1, confirmed by hand on
2026-09-09 against grafana/loki:3.7.7 (`/loki/api/v1/status/buildinfo`
reports version 3.7.7, revision 7a40404f, branch release-3.7.x). Both
encodings are refused:

$ curl -X POST localhost:3100/otlp/v1/logs \
-H 'Content-Type: application/json' --data '{"resourceLogs":[]}'
9error at least one valid stream is required for ingestion
HTTP 422

$ curl -X POST localhost:3100/otlp/v1/logs \
-H 'Content-Type: application/x-protobuf' --data-binary ''
9error at least one valid stream is required for ingestion
HTTP 422

Control, issued against the same container at the same moment: one valid
record with a `service.name` resource attribute and a string body returned
HTTP 204. So the container was healthy and accepting, and the refusal is
specific to the empty export rather than a startup or readiness artefact.

This is the second store to refuse an empty export, and it is a regression
against the same clause Quickwit fails: OTLP's `docs/specification.md` says
"If the server receives an empty request (a request that does not carry any
telemetry data) the server SHOULD respond with success." Unlike Quickwit's
500 it is a 4xx, so a Collector will not retry it indefinitely — the failure
is loud rather than a retry storm — but conformant traffic is still refused,
and a Collector flushing on a timer with nothing to send produces a steady
stream of 422s with no user-actionable cause.

Recorded separately: the error body is returned protobuf-framed (the leading
`9` is a length prefix, not text) even when the request carried
`Content-Type: application/json`, so a JSON client cannot read the reason it
was refused. Same class as the OpenObserve response-encoding draft.
6 changes: 5 additions & 1 deletion cases/otlp-logs/record-without-body.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ rule:
a reader will take it for the event's.

observed:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11525
- https://github.com/VictoriaMetrics/VictoriaLogs/issues/1778

send:
format: otlp-json
Expand Down Expand Up @@ -52,3 +52,7 @@ notes: >
substitution rather than an accident of one payload. Nothing errors at
ingest, and a dashboard renders the notice as though the application had
logged it.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/20, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
11 changes: 11 additions & 0 deletions cases/otlp-logs/timestamp-nanosecond-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,17 @@ notes: >
does not give one. Recorded, not adjudicated — but this is now a strong
question to put to the specification's owners rather than a curiosity.

Loki 3.7.7 re-measured 2026-09-09, after the adapter was bumped from 3.1.1:
sent "1788968196123456789", read back "1788968196123456789" — still a string
of nanoseconds, still every digit. The behaviour is unchanged across a minor
release, which is worth having: it makes the Loki row evidence about a
deliberate choice rather than one build's accident. Note the published
2026-09-08 page recorded this column as `release-3.1.x-89fe788`, the string
that Loki's buildinfo returned then; 3.7.7 returns a plain "3.7.7". Any table
quoting "Loki 3.1.1" for this case — including adjudication issue #12 as
first written — is citing the older measurement and should be updated to
name both.

Adjudication opened 2026-09-09:
https://github.com/DeviousCardi/specmatrix/issues/12, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the OpenTelemetry
Expand Down
2 changes: 1 addition & 1 deletion cases/otlp-logs/timestamp-outside-ingest-window.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ rule:
notice the client cannot read, whether that is an empty partial_success or
one encoded in a format the request did not use.
observed:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11526
- https://github.com/VictoriaMetrics/VictoriaLogs/issues/1777
- https://github.com/openobserve/openobserve/issues/14293
- docs/ROADMAP.md, 0.1 result, OpenObserve v0.92.2 at default settings

Expand Down
4 changes: 4 additions & 0 deletions cases/otlp-logs/timestamp-zero.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,7 @@ notes: >
Quickwit reports one rejected record in partial_success, so it is not silent
here — but that report is protobuf under a JSON content type, so a conformant
client never sees it.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/21, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-logs/trace-id-all-zero.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,7 @@ notes: >
store to discard it — but a query for records belonging to trace
00000000000000000000000000000000 returns every untraced record on Parseable
and none on the others, with no error raised on either side.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/22, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-metrics/data-point-without-attributes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,7 @@ notes: >
sees the previous run's point too. That is acceptable here because the
assertion is that the metric exists at all, and it is why this row is
`present` rather than `exact`.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/23, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-metrics/exponential-histogram.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,7 @@ notes: >
stabilisation and deferred follow-up; this looks like part of that work
rather than something unreported, so it is linked and recorded instead of
filed again.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/24, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
1 change: 1 addition & 0 deletions cases/otlp-metrics/gauge-negative-zero.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ rule:
sent as -0.0 has changed the value it was given.
observed:
- https://github.com/grafana/mimir/issues/16539
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/11533

send:
format: otlp-metrics-json
Expand Down
4 changes: 4 additions & 0 deletions cases/otlp-metrics/histogram-inf-bound.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,7 @@ notes: >
The Prometheus translation should produce a _bucket series with le="+Inf"
holding 3; a store that produced le="0.1" only has dropped the implicit
bucket.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/25, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-metrics/histogram-no-buckets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,7 @@ notes: >
read and only this data point was discarded — along with its count and sum,
which have nothing to do with the missing buckets. 200, no counter, no log.
Filed as VictoriaMetrics#11528.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/26, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-metrics/sum-int64-max.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,7 @@ notes: >
Recorded rather than judged, like remote-write/counter-u64-max. The loss is a
property of a float-valued store and not a fault; the row exists so a reader
can see whether the stores agree on the rounded value.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/27, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
4 changes: 4 additions & 0 deletions cases/otlp-metrics/unit-suffix-on-metric-name.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,7 @@ notes: >
GreptimeDB is N/A rather than PASS only because its PromQL requires a metric
matcher in every selector, so the question cannot be put through that API.
`show tables` answers it: the suffix is appended.

Adjudication opened 2026-09-09: https://github.com/DeviousCardi/specmatrix/issues/28, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the specification's
owners — this check stays `present` until that answer arrives.
Loading
Loading