0.4 — OTLP traces: Jaeger, Tempo, Quickwit, OpenObserve

AGENTS.md

@@ -143,7 +143,12 @@ runner encodes it to whatever the backend accepts. `send.format` names the
 format the file is written in, `send.encodings` lists what it may be sent as,
 and an adapter's `formats:` says what it accepts on the wire.
 
-- `otlp-json` — real OTLP JSON, sendable with `curl` as it stands.
+- `otlp-json` — real OTLP JSON, sendable with `curl` as it stands. Logs by
+  default; `otlp-metrics-json` and `otlp-traces-json` are the same convention
+  for the other two signals, each with its own protobuf counterpart
+  (`otlp-metrics-protobuf`, `otlp-traces-protobuf`) — three formats rather than
+  one because the three OTLP export services are different messages on the
+  wire, not a shared envelope.
 - `es-ndjson` — a real `_bulk` body.
 - `loki-json` — the push body as Loki itself documents it:
   `streams[].stream` for labels and `streams[].values` as `[timestamp_ns, line]`
@@ -222,6 +227,23 @@ a metric named `a.b.c` cannot be named in a query even though the data is there.
 It is **never** a way to exclude a case the store fails. If you are reaching for
 it because a verdict is inconvenient, you are writing a false column.
 
+Two `container:` fields exist for settings a CLI flag cannot reach:
+
+- `extra_ports:` publishes additional container ports, for a store that splits
+  ingest and query across two ports of one container — Jaeger's OTLP receiver
+  and its own query API, Tempo's OTLP receiver and its `/ready`/query API. A
+  `readback.request` or `container.ready.request` naming an absolute URL
+  (`http://` or `https://`) is sent to that URL verbatim rather than through
+  `container.port`'s base URL.
+- `config:` is an inline file this project's own `specmatrix up` writes to a
+  temp path and mounts at a fixed path inside the container
+  (`/etc/specmatrix/config.yaml`), for a required setting with no CLI-flag
+  equivalent at all — confirm that by reading the binary's own `-help` first,
+  not by assuming. Tempo's OTLP receiver binds to `127.0.0.1` inside the
+  container unless its config sets an explicit `0.0.0.0` endpoint, and nothing
+  in the API says so — only `docker logs` names the bound address, and every
+  ingest from outside answers a bare connection reset with no HTTP status.
+
 ## Reporting a divergence
 
 Findings are worth more filed than tabulated.

Cargo.toml

@@ -18,7 +18,7 @@ include = ["src/**", "cases/**", "backends/**", "README.md", "LICENSE", "NOTICE"
 anyhow = "1"
 chrono = "0.4"
 clap = { version = "4", features = ["derive"] }
-opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic-messages", "logs", "metrics", "with-serde"] }
+opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic-messages", "logs", "metrics", "trace", "with-serde"] }
 prost = "0.14"
 rand = "0.9"
 regex = "1"

README.md

@@ -86,7 +86,7 @@ much longer to build and is what makes the results worth citing.
 
 ## Status
 
-Not yet published. Five protocols, ten backends and sixty-six checks run
+Not yet published. Six protocols, twelve backends and eighty-one checks run
 unattended from one command; the write-up that has to precede publication is
 not done.
 
@@ -123,6 +123,20 @@ returns for that record. And OpenObserve's push landed at
 request's example used, confirmed only by trying both against a running
 container rather than trusting the documentation.
 
+OTLP traces followed, across Jaeger, Grafana Tempo, Quickwit and OpenObserve —
+fifteen checks, none sharing a read-back shape: Tempo alone echoes OTLP's own
+span JSON (with trace and span IDs coming back base64, protobuf JSON's own
+encoding for a bytes field, so those two fields compare as bytes rather than as
+equal strings); Jaeger answers only its own query model, where a span event
+becomes a `logs[]` entry and a timestamp is microseconds rather than OTLP's
+nanoseconds; Quickwit flattens spans into search columns; and OpenObserve does
+the same but keeps a span's events as a JSON string rather than structured
+JSON, unreachable by a field pointer at all. A span event's timestamp and name
+are recorded as `present` rather than `exact` for this reason — no store here
+keeps OTLP's own shape for it, which makes the check a record of where each
+one put the data rather than a pass/fail on a shape none of them chose to
+keep.
+
 Not everything that differs is a finding, and the corpus says so. A NaN sample
 dropped at ingest, an exponential histogram with no representation, a metric
 name that does or does not gain its unit as a suffix: those are recorded with
@@ -140,7 +154,7 @@ was confirmed.
 Rust and Docker; nothing else.
 
 ```sh
-cargo test                        # 166 tests, no network, no containers
+cargo test                        # 183 tests, no network, no containers
 
 cargo run -- up   --backend loki  # start a backend from its adapter
 cargo run -- run  --backend loki --suite otlp-logs
@@ -153,6 +167,9 @@ cargo run -- matrix --suite otlp-logs --manage \
 cargo run -- matrix --suite remote-write --manage \
   --backends prometheus,mimir,greptimedb,victoriametrics
 
+cargo run -- matrix --suite otlp-traces --manage \
+  --backends jaeger,tempo,quickwit,openobserve
+
 # the bytes a backend actually receives, for reproducing a finding by hand
 cargo run -- encode cases/remote-write/minimal-gauge.json \
   --from remote-write-json --to remote-write-protobuf --out body.snappy

backends/jaeger.yaml

@@ -0,0 +1,67 @@
+# Adapter for Jaeger, all-in-one.
+#
+# Confirmed by hand against jaegertracing/jaeger:2.20.0 on 2026-09-09.
+#
+# The OTLP HTTP receiver is on 4318 and needs no flag in Jaeger 2.x — v2 is a
+# rewrite around the OpenTelemetry Collector and takes OTLP by default, unlike
+# Jaeger 1.x which needed COLLECTOR_OTLP_ENABLED=true.
+#
+# Read-back is Jaeger's own query API, not OTLP — there is no way to ask
+# Jaeger for a span in the shape it was sent, only in Jaeger's model. What had
+# to be confirmed by hand rather than assumed:
+#
+#   GET /api/traces/<trace id, hex, no dashes>
+#   {"data":[{"spans":[{"operationName":"...","startTime":<int, MICROSECONDS>,
+#     "duration":<int, microseconds>,"tags":[{"key":...,"type":...,"value":...}],
+#     "logs":[...], ...}]}]}
+#
+# `startTime` is microseconds, not the nanoseconds OTLP sends — a precision
+# check on this column is checking Jaeger's own model, not a store that failed
+# to keep OTLP's precision; nanosecond-precision checks are `present`, not
+# `exact`, for this reason and this reason only.
+
+name: jaeger
+container:
+  image: jaegertracing/jaeger:2.20.0
+  port: 4318
+  extra_ports: [16686]
+  env:
+    COLLECTOR_OTLP_ENABLED: "true"
+  ready:
+    # The query UI, not the ingest port, which has no unauthenticated health
+    # endpoint of its own — and the query port is what a case's read-back
+    # actually reads from, so this is also confirming the port a case needs
+    # is the one that is up.
+    request: GET http://localhost:16686/
+    expect_status: 200
+
+auth:
+  kind: none
+
+protocols:
+  otlp-traces:
+    formats: [otlp-traces-json, otlp-traces-protobuf]
+    ingest:
+      request: POST /v1/traces
+    readback:
+      # A different port than ingest. The adapter's base URL is the ingest
+      # port (4318, matched to `container.port` so `specmatrix up` reports the
+      # right address); read-back overrides host and port outright.
+      request: GET http://localhost:16686/api/traces/{{ trace_id_hex }}
+      records: /data/0/spans
+      fields:
+        name: /operationName
+        startTimeUnixNano: /startTime
+        traceId: /traceID
+        spanId: /spanID
+        # A span event becomes a Jaeger "log", confirmed by hand on
+        # 2026-09-09: {"logs":[{"timestamp":<int, MICROSECONDS>,
+        # "fields":[{"key":"event","type":"string","value":<the event name>}]}]}.
+        # Two renames and a precision drop packed into one mapping — there is
+        # no field in Jaeger's model that is simply "the event's timestamp" or
+        # "the event's name" at a fixed pointer, this is where they live.
+        events.0.timeUnixNano: /logs/0/timestamp
+        events.0.name: /logs/0/fields/0/value
+      poll:
+        interval_ms: 1000
+        timeout_ms: 30000

backends/openobserve.yaml

@@ -104,6 +104,36 @@ protocols:
         interval_ms: 1000
         timeout_ms: 30000
 
+  # Confirmed by hand on 2026-09-09. Traces land in a stream named `default`,
+  # same as the loki-push protocol above, and answer the same `type=traces`
+  # search convention. `span_id` and `trace_id` keep their hex spelling;
+  # `name` becomes `operation_name`; `start_time`/`end_time` are nanoseconds,
+  # unlike the microseconds this adapter's OTLP-logs `_timestamp` column
+  # carries — traces keep full precision where logs do not.
+  otlp-traces:
+    formats: [otlp-traces-json]
+    ingest:
+      request: POST /api/default/v1/traces
+      headers:
+        Content-Type: application/json
+    readback:
+      request: POST /api/default/_search?type=traces
+      body:
+        query:
+          sql: "SELECT * FROM \"default\" WHERE trace_id = '{{ trace_id_hex }}'"
+          start_time: "{{ window_start_us }}"
+          end_time: "{{ window_end_us }}"
+      records: /hits
+      fields:
+        name: /operation_name
+        traceId: /trace_id
+        spanId: /span_id
+        startTimeUnixNano: /start_time
+        endTimeUnixNano: /end_time
+      poll:
+        interval_ms: 1000
+        timeout_ms: 30000
+
   loki-push:
     formats: [loki-json]
     # Confirmed by hand on 2026-09-09 against v0.92.2. openobserve/openobserve

backends/quickwit.yaml

@@ -130,6 +130,41 @@ protocols:
   # to wait longer, not to lower the store's commit timeout: waiting changes
   # nothing about what Quickwit does, and tuning it would be the corpus
   # arranging its own result.
+  # Confirmed by hand on 2026-09-09. Quickwit creates otel-traces-v0_7 itself
+  # at start-up, the same way it creates the logs index; protobuf only, same
+  # as the OTLP logs endpoint. Read-back field names are Quickwit's own
+  # flattened columns rather than the OTLP JSON names.
+  otlp-traces:
+    formats: [otlp-traces-protobuf]
+    setup_verify:
+      request: GET /api/v1/indexes/otel-traces-v0_7
+      poll:
+        interval_ms: 500
+        timeout_ms: 60000
+    ingest:
+      request: POST /api/v1/otlp/v1/traces
+      headers:
+        Content-Type: application/x-protobuf
+    readback:
+      request: GET /api/v1/otel-traces-v0_7/search?query=trace_id:{{ trace_id_hex }}&max_hits=10
+      records: /hits
+      fields:
+        name: /span_name
+        traceId: /trace_id
+        spanId: /span_id
+        startTimeUnixNano: /span_start_timestamp_nanos
+        endTimeUnixNano: /span_end_timestamp_nanos
+        # Confirmed by hand on 2026-09-09: an event becomes one entry of a
+        # top-level `events` array, `{"event_name":...,"event_timestamp_nanos":...}`
+        # — renamed from OTLP's own `name`/`timeUnixNano`, full nanosecond
+        # precision kept.
+        events.0.timeUnixNano: /events/0/event_timestamp_nanos
+        events.0.name: /events/0/event_name
+      poll:
+        interval_ms: 1000
+        timeout_ms: 30000
+
+
   es-bulk:
     formats: [es-ndjson]
     # Quickwit will not create an index on write, so this protocol creates one

backends/tempo.yaml

@@ -0,0 +1,82 @@
+# Adapter for Grafana Tempo, single-binary local storage.
+#
+# Confirmed by hand against grafana/tempo:3.0.3 on 2026-09-09.
+#
+# Two settings below have no CLI-flag equivalent — confirmed against the
+# binary's own -help, which exposes -storage.trace.backend but nothing for
+# the receiver — so they come from `container.config`, an inline file this
+# project's own `specmatrix up` mounts, rather than a flag on `command`.
+#
+# The OTLP receiver's `endpoint: 0.0.0.0:4318` matters and is not
+# boilerplate: without it, Tempo starts the receiver bound to 127.0.0.1 only,
+# inside the container, and every ingest from outside answers a bare
+# connection reset with no HTTP status at all. `docker logs` names the bound
+# address; nothing in the API does, and no error is more misleading than
+# "the connection reset" for a value it never validated against.
+#
+# Read-back answers real OTLP JSON, unlike Jaeger's own model — batches,
+# scopeSpans, spans, the same shape a case sends — except trace_id and span_id
+# come back base64, which is protobuf JSON's own encoding for a bytes field
+# and not a Tempo rewrite; the crate this project encodes with uses a hex
+# serializer for the same fields instead, so those two fields need comparing
+# as bytes, not as equal strings — a check on them is `present`, not `exact`.
+#
+# `/api/traces/<id>` answers 404 until the trace has flushed from the
+# ingester, which local storage does quickly; `ready.settle_ms` covers the gap
+# between the query API accepting connections and a trace sent at that moment
+# actually flushing.
+
+name: tempo
+version_from:
+  request: GET http://localhost:3200/api/status/buildinfo
+  field: version
+container:
+  image: grafana/tempo:3.0.3
+  port: 4318
+  extra_ports: [3200]
+  command: ["-config.file=/etc/specmatrix/config.yaml"]
+  config: |
+    server:
+      http_listen_port: 3200
+    distributor:
+      receivers:
+        otlp:
+          protocols:
+            http:
+              endpoint: 0.0.0.0:4318
+    storage:
+      trace:
+        backend: local
+        local:
+          path: /var/tempo/traces
+        wal:
+          path: /var/tempo/wal
+  ready:
+    request: GET http://localhost:3200/ready
+    expect_status: 200
+    # Measured on 2026-09-09: /ready answers within a couple of seconds but a
+    # trace sent in the same moment can still 404 for several more while the
+    # ingester finishes starting. Margin over that gap, not a tuning of what
+    # gets measured.
+    settle_ms: 5000
+
+auth:
+  kind: none
+
+protocols:
+  otlp-traces:
+    formats: [otlp-traces-json, otlp-traces-protobuf]
+    ingest:
+      request: POST /v1/traces
+    readback:
+      request: GET http://localhost:3200/api/traces/{{ trace_id_hex }}
+      records: /batches/0/scopeSpans/0/spans
+      fields:
+        name: /name
+        traceId: /traceId
+        spanId: /spanId
+        startTimeUnixNano: /startTimeUnixNano
+        endTimeUnixNano: /endTimeUnixNano
+      poll:
+        interval_ms: 1000
+        timeout_ms: 30000

cases/otlp-traces/attribute-int64-max.json

@@ -0,0 +1,4 @@
+{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"specmatrix"}}]},
+"scopeSpans":[{"scope":{"name":"specmatrix"},"spans":[{"traceId":"{{ trace_id_hex }}","spanId":"0902030405060708",
+"name":"big-attribute","kind":1,"startTimeUnixNano":"{{ now_ns }}","endTimeUnixNano":"{{ now_ns_fractional }}",
+"attributes":[{"key":"specmatrix.bignum","value":{"intValue":"9223372036854775807"}}]}]}]}]}

cases/otlp-traces/attribute-int64-max.yaml

@@ -0,0 +1,30 @@
+id: otlp-traces/attribute-int64-max
+protocol: otlp-traces
+title: A span attribute at the largest int64
+
+rule:
+  basis: spec
+  spec: opentelemetry-proto/common/v1
+  section: AnyValue.int_value
+  text: >
+    int_value is an int64. OTLP's JSON mapping writes it as a string because
+    a JSON number cannot hold every int64 exactly, and a store whose
+    attribute storage is a JSON number or a float cannot represent this value
+    without loss.
+
+send:
+  format: otlp-traces-json
+  encodings: [otlp-traces-json, otlp-traces-protobuf]
+  body: cases/otlp-traces/attribute-int64-max.json
+
+expect:
+  ingest: accepted
+  readback:
+    match: present
+    on: [attributes.specmatrix.bignum]
+
+notes: >
+  Recorded rather than judged, like the equivalent metrics and remote-write
+  cases: any loss here is a property of a store's attribute storage, not a
+  conformance failure by itself, and the row exists so a reader can see
+  whether it agrees with the other columns.

cases/otlp-traces/dropped-attributes-count-nonzero.json

@@ -0,0 +1,4 @@
+{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"specmatrix"}}]},
+"scopeSpans":[{"scope":{"name":"specmatrix"},"spans":[{"traceId":"{{ trace_id_hex }}","spanId":"0c02030405060708",
+"name":"dropped-attrs","kind":1,"startTimeUnixNano":"{{ now_ns }}","endTimeUnixNano":"{{ now_ns_fractional }}",
+"droppedAttributesCount":7}]}]}]}