Skip to content

0.4 — OTLP traces: Jaeger, Tempo, Quickwit, OpenObserve - #9

Merged
DeviousCardi merged 1 commit into
mainfrom
otlp-traces
Sep 9, 2026
Merged

DeviousCardi merged 1 commit into
mainfrom
otlp-traces

Conversation

@DeviousCardi

Copy link
Copy Markdown
Owner

Summary

Part G of PLAN-1.0.md: a new otlp-traces suite (15 checks) run against four
backends — Jaeger, Grafana Tempo, Quickwit, OpenObserve — each of which
represents a span in a genuinely different shape than OTLP's own.

All 15 cases pass on all 4 columns (60/60). 183 unit tests, 0 clippy warnings.

Runner changes (generalize past traces)

  • Absolute-URL support in Runner::send() and docker::wait_ready(): a
    rendered request path starting with http:///https:// is used verbatim,
    bypassing base_url. Needed because Jaeger and Tempo split ingest and
    query across two ports of one container. Regression-tested with two
    separate stub servers so neither endpoint can fall back to the other.
  • Container.extra_ports / Container.config: publish additional
    container ports, and mount an inline config file at a fixed path, for
    settings with no CLI-flag equivalent. Tempo needs this — its OTLP receiver
    binds to 127.0.0.1 inside the container unless the config sets an
    explicit 0.0.0.0 endpoint, confirmed only via docker logs, not the API.
  • export_report() made protocol-aware: it was comparing
    request_encoding == "otlp-json" literally, silently wrong for every OTLP
    metrics/traces JSON export, and its protobuf decode path was hardcoded to
    the logs response type — a genuine traces/metrics partial-success report
    decoded to None and was invisibly dropped before this fix.
  • field_of() fallback chain extended: attributes.<key> searches the
    record's own OTLP-shaped attributes array; any other dotted field with no
    adapter mapping is tried as a JSON pointer.

A regression found and fixed along the way

An earlier edit meant to add trace_id_base64 to vars_for() silently
failed to save (a multi-assert Python script whose last assert failed, so the
final file write never ran) while a separately-applied edit to read_back()
for the same feature did save — masking the omission. The always-empty
trace_id_base64, combined with first_record()'s .contains(run_key)
check ("".contains("") is true in Rust), spuriously matched the first
record in any response. Fixed by inserting the missing variable and adding a
permanent guard: first_record() now refuses to match an empty key, with a
regression test.

Adapter notes (confirmed against running containers)

  • Tempo answers real OTLP JSON on read-back, except trace/span IDs come
    back base64 (protobuf JSON's bytes-field encoding) where this project's
    encoder uses hex — compared as bytes, not equal strings.
  • Jaeger answers only its own query model: microsecond timestamps, and a
    span event becomes a logs[] entry.
  • Quickwit and OpenObserve flatten spans into search columns;
    OpenObserve stores events as a JSON string, unreachable by any field
    pointer.

event-timestamp-outside-span is match: present rather than exact
because no store here keeps OTLP's own field shape for a span event — the
check records where each one put the data.

🤖 Generated with Claude Code

https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK

…(Part G)

Fifteen checks in a new `otlp-traces` suite, run against four backends that
each represent a span in a different shape than OTLP's own — which forced
several runner fixes that generalize past traces:

- `Runner::send()` and `docker::wait_ready()` now treat a rendered request
  path starting with `http://`/`https://` as an absolute URL and use it
  verbatim, bypassing `base_url`. Needed because Jaeger and Tempo split
  ingest and query across two ports of one container; regression-tested with
  two separate stub servers so neither endpoint can silently fall back to
  the other.
- `Container` gained `extra_ports` (publish additional container ports) and
  `config` (an inline file `specmatrix up` writes to a temp path and mounts
  at a fixed path), for settings with no CLI-flag equivalent — confirmed by
  reading each binary's own `-help` first. Tempo needs this: its OTLP
  receiver binds to 127.0.0.1 inside the container unless the config sets an
  explicit 0.0.0.0 endpoint, and nothing in the API says so, only
  `docker logs`.
- `export_report()` was comparing `request_encoding == "otlp-json"` literally,
  so it was silently wrong for every OTLP metrics/traces JSON export
  (`otlp-metrics-json`/`otlp-traces-json` never matched) and its protobuf
  decode path was hardcoded to the logs response type — a genuine traces or
  metrics partial-success report decoded to `None` and was invisibly
  dropped. Now takes `protocol: &str` and branches field names and decode
  type per protocol; confirmed against Jaeger, and against Quickwit's own
  genuine json-response-to-protobuf-request mismatch, which survives
  correctly.
- `field_of()` gained a fallback chain for fields with no adapter mapping:
  `attributes.<key>` searches the record's own OTLP-shaped attributes array,
  and any other dotted field with no mapping is tried as a JSON pointer.
  Verified against Tempo, isolating cases via move-aside/restore to confirm
  each fix in a clean run.

Found and fixed a real regression along the way: an earlier edit meant to add
`trace_id_base64` to `vars_for()` silently failed to save (a multi-assert
Python script whose last assert failed, so the final write never ran), while
a separately-applied edit to `read_back()` for the same feature did save —
masking the omission. Combined with `first_record()`'s `.contains(run_key)`
check, where `"".contains("")` is `true` in Rust, the always-empty
`trace_id_base64` spuriously matched the first record in any response,
breaking `a_leftover_record_from_another_run_does_not_count`. Fixed by
inserting the missing variable and adding a permanent guard —
`first_record()` now refuses to match an empty key — plus a regression test.

Adapter notes, each confirmed against a running container:

- Tempo answers real OTLP JSON on read-back, unlike every other store here,
  except trace/span IDs come back base64 (protobuf JSON's own bytes-field
  encoding) where this project's encoder uses hex — compared as bytes, not
  equal strings.
- Jaeger answers only its own query model: `startTime`/`duration` in
  microseconds, not OTLP's nanoseconds, and a span event becomes a `logs[]`
  entry keyed by `fields[].key == "event"`.
- Quickwit and OpenObserve both flatten spans into search columns; OpenObserve
  stores a span's events as a JSON string rather than structured JSON, so no
  JSON-pointer mapping can reach it at all.

`event-timestamp-outside-span` is `match: present` rather than `exact` for
this reason: no store here keeps OTLP's own field shape for a span event, so
the check records where each one put the data rather than judging a shape
none of them chose to keep.

All 15 cases pass on all 4 columns (60/60). 183 unit tests, 0 clippy warnings.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018k65nFSzuwsHwYnpSHNaeK
@DeviousCardi
DeviousCardi merged commit e47ea02 into main Sep 9, 2026
3 checks passed
@DeviousCardi
DeviousCardi deleted the otlp-traces branch September 9, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant