Close out the 1.0 checklist: versions confirmed, every divergence filed, every present case adjudicated - #46
Merged
Merged
Conversation
The adapters for Prometheus, Mimir and GreptimeDB each pasted a confirmation transcript from an older image than the tag they pin. The tags and the published matrices were right — `backend_version` comes from the live version probe in `src/runner.rs`, not from the tag — so this was a documentation error, not a data-integrity one, but it read as though three columns had been measured against software they were not measured against. Re-probed against the pinned tags on 2026-09-09 and pasted what came back: prom/prometheus:v3.14.0 /api/v1/status/buildinfo -> 3.14.0 grafana/mimir:3.2.0 /prometheus/api/v1/status/buildinfo -> 3.2.0 greptime/greptimedb:v1.2.0 /status -> 1.2.0 GreptimeDB's `/status` no longer carries `source_time`; the field was on 0.17.2 and is gone on 1.2.0. Noted in the adapter, since the pointer the adapter uses is `/version` and is unaffected. Loki's adapter had moved to 3.7.7 while the published otlp-logs page was measured against 3.1.x, so the suite was re-run against 3.7.7. Two results: - `timestamp-nanosecond-precision` still returns every digit. The behaviour is unchanged across a minor release, which makes the Loki row in adjudication #12 evidence about a deliberate choice rather than one build's accident. Recorded that any table quoting "Loki 3.1.1" for that case is citing the older measurement. - `empty-batch` changed from PASS to REJECT: 3.7.7 answers 422 "at least one valid stream is required for ingestion" to an export carrying no records, where 3.1.1 answered 204. Confirmed by hand in both encodings, with a control — one valid record against the same container at the same moment returned 204, so the refusal is specific to the empty export. This is the second store to fail the clause OTLP states as "if the server receives an empty request ... the server SHOULD respond with success"; unlike Quickwit's 500 it is a 4xx, so it costs error-log noise on quiet services rather than a retry storm. README: the check count said eighty-one; `tools/check_corpus.py` counts eighty. And it claimed every finding "is filed with the project it concerns", which is not true of any of them yet — the drafts exist and none has been filed, which is itself the reason there is no published page. Corrected to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Jaeger column published a blank version for the whole otlp-traces suite:
`backends/jaeger.yaml` had no `version_from` block at all, and nothing caught
it, because the adapter audit only checked that the image was pinned.
Jaeger 2.x is an OpenTelemetry Collector distribution and has no version
endpoint on either port it exposes here — `/api/version` and `/api/v3/version`
answer 404, and `/version` is swallowed by the UI's single-page-app catch-all
and returns HTML. The version is on the Collector's internal telemetry
endpoint instead, so 8888 joins 16686 in `extra_ports` and the version is read
with a pattern, the way VictoriaLogs already is:
$ curl -s localhost:8888/metrics | grep '^target_info'
target_info{service_instance_id="8e20555c-...",service_name="jaeger",
service_version="v2.20.0"} 1
Confirmed end to end: `run --backend jaeger --suite otlp-traces` now heads its
output `jaeger v2.20.0`, and all fifteen checks still pass.
Added `every_committed_adapter_can_report_a_version` so this cannot recur. It
asserts a `version_from` with either a `field` or a `pattern` for every adapter
in `backends/`, and all twelve pass.
Found while re-confirming columns for the 1.0 checklist, which asks for twelve
columns each carrying a version. Worth recording how it surfaced: a local
`target/release/specmatrix` built before `extra_ports` existed silently started
Jaeger without publishing 16686, so `up` never went ready. The stale binary was
the reason the port was missing, but chasing it is what exposed the missing
version probe, which was real. Every suite result taken today was re-run
against a freshly built binary; the Loki otlp-logs verdicts are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Part K asks that every `ALTER` be filed upstream and linked from the check. Audited per (case, backend) pair rather than per case, which is the level the requirement is actually about: 23 pairs, 15 already filed, 8 outstanding. All eight were reproduced by hand against a running container with a control before anything was written, and they did not all turn out to be defects. Filed: - VictoriaMetrics#11533 — an OTLP gauge of -0.0 is stored as +0.0. The sign bit is verified present in the encoded protobuf (`21 00..00 80`), and the discriminator is division, not equality, since -0.0 == 0.0 is true: `1/x` reads `+Inf` on VictoriaMetrics and `-Inf` on Prometheus 3.14.0 given the same body in the same minute. - VictoriaLogs#1780 — a log line containing invalid UTF-8 is stored *intact*, which is better than the reference manages, and then written raw into the JSON query response, so the whole response fails UTF-8 decoding and every other record in it becomes unreadable. Filed against VictoriaLogs, not VictoriaMetrics: the tracker was split, which this pass also discovered. - OpenObserve#14344 — the Loki push API stores a nanosecond timestamp truncated to microseconds (789 ns lost, string to number). Loki 3.7.7 and VictoriaLogs 1.52.0 both keep all nine digits through the same API, so the precision is plainly keepable here. Kept deliberately separate from the OTLP version of the same truncation, which is under adjudication (#12) and has no rule to cite. - OpenObserve#14345 — the push API answers 204 with an empty body to a line outside the 5-hour ingest window and discards it, while the same server's OTLP endpoint discards it *and says so*, naming ZO_INGEST_ALLOWED_UPTO. The store's own better endpoint is the control. Reproduced and commented rather than filed anew: VictoriaLogs#1777 already covers the silent retention drop through OTLP, and the push API is the same defect on a second endpoint, so it went on that issue with its own control. Deliberately not filed, recorded in each case's `notes:` so the page can be read without re-deriving it: - `loki-push/empty-push` on VictoriaLogs, and `loki-push/label-name-invalid` on VictoriaLogs and OpenObserve. All three accept a payload Loki refuses, and in both label cases the label is kept intact and queryable — verified, the stored record carries `"1zone":"a"`. What a sender loses is a validation signal, not data. Filing that against two projects would be noise, and neither claims strict Loki parity. Relinked: the two VictoriaLogs issues were transferred out of the VictoriaMetrics tracker, so #11525 and #11526 are now VictoriaLogs #1778 and #1777. Three cases cited the old numbers. One issue in this pass was filed into the wrong tracker for the same reason and was refiled — VictoriaMetrics#11534 is closed pointing at VictoriaLogs#1780. 23 pairs: 20 filed and linked, 3 reasoned, 0 unlinked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ADJUDICATION.md` is explicit that a `present` check does not sit at `present` by default — "every one gets an adjudication issue, even if the issue stays open for years waiting on an answer" — and `PLAN-1.0.md` makes it a release requirement. One of the thirty-three had one. Opened the other thirty-two (#14–#45), one per check, each from the `Adjudication` template and each carrying: - what every implementation that has run the check did, with its version, taken from the published matrices rather than retyped; - the clause the check cites, quoted from the check's own `rule.text`, and where it stops; - an explicit note that the upstream question has not been put yet, because filing with a specification's owners needs approval first — so the issue stays open with nothing to promote, which is what `ADJUDICATION.md` step 3 says such an issue is for. Each issue is linked from its check's `notes:`, the way `#12` already was, so a reader of the corpus can get from a `present` row to the reason it is `present` without leaving the file. Jaeger's rows in eight of these carry the pinned image tag with a footnote rather than a probed version: the published traces run predates the version probe added in ae9d47f, so the matrix recorded a blank. Marked as the tag rather than silently presented as a measurement. 33 of 33 `present` checks now carry an adjudication issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g on The CI example pinned `@v1.0.0`, which resolves to nothing — the same class of error as the dogfood run that 404'd on `latest` before any release existed. It names `v0.9.0`, which is what is being tagged. Status rewritten because the honest blocker changed. It said the write-up was outstanding; the write-ups are done, the divergences are filed or reasoned, and every `present` check carries an adjudication issue. What remains is that a quarterly cadence cannot be demonstrated inside a week. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DeviousCardi
force-pushed
the
evidence-corrections
branch
from
September 9, 2026 16:45
8f42963 to
e816c96
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Works through
PLAN-1.0.md's Part K checklist. Five commits, each standalone.What was wrong, and is now right
Three adapters pasted a confirmation transcript from an older image than the
tag they pin — Prometheus showed 3.6.0 under
v3.14.0, Mimir 2.17.1 under3.2.0, GreptimeDB 0.17.2 underv1.2.0. Re-probed all three against thepinned tags: the tags and the published matrices were right (
backend_versioncomes from the live probe in
src/runner.rs, not the tag), and the commentswere stale. Fixed with the real output.
The Jaeger column published no version at all.
backends/jaeger.yamlhadno
version_fromblock. Jaeger 2.x is a Collector distribution with no versionendpoint on either port it exposes, so the version now comes from the
Collector's telemetry endpoint (
target_info{service_version=...}), read with apattern the way VictoriaLogs already is.
every_committed_adapter_can_report_a_versionnow asserts a version source for all twelve, because the audit only checked
image pinning and this had slipped through.
Loki had moved to 3.7.7 while the published
otlp-logspage was measuredagainst 3.1.x. Re-ran it.
timestamp-nanosecond-precisionstill keeps everydigit, which strengthens adjudication #12;
empty-batchchanged from PASS toREJECT — 3.7.7 answers 422 to an export carrying no records where 3.1.1
answered 204, against OTLP's "the server SHOULD respond with success".
Reproduced by hand in both encodings with a control.
README claimed every finding was filed. Audited per (case, backend) pair:
23 ALTER pairs, 15 filed. The eight outstanding were each reproduced against a
running container with a control, and did not all turn out to be defects.
Filed upstream in this pass
otlp-metrics/gauge-negative-zero—1/xis+Infwhere Prometheus gives-Infloki-push/line-invalid-utf8— bytes stored intact, then written raw into a JSON responseloki-push/timestamp-nanosecond-precision— 789 ns droppedloki-push/timestamp-older-than-window— silent on the push API, reported on OTLPThree are deliberately not filed with the reasoning in each case's
notes:— accepting a payload Loki refuses, with the data kept intact and queryable, is
laxness rather than a defect.
Two links were repointed: the VictoriaLogs issues were transferred out of the
VictoriaMetrics tracker, so
#11525/#11526are now#1778/#1777. One issuein this pass was filed into the wrong tracker for the same reason and refiled.
Adjudication
Opened #14–#45, one per
presentcheck, each carrying what everyimplementation did, the clause it cites, and an explicit note that the upstream
question has not been put yet. 33 of 33 now carry one, up from 1.
Part K after this
presentcase has an adjudication issuev0.9.0to be tagged after this mergesThe last item is why this does not tag
v1.0.0. A quarterly cadence cannot bedemonstrated inside a week;
v0.9.0is the corpus as it stands.🤖 Generated with Claude Code