Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions ADJUDICATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Adjudication

`AGENTS.md`'s "Adjudication" section gives the order of precedence a check's
verdict rests on:

1. **The specification says so.** Cite the clause. Done — a check written
this way never needs what follows.
2. **The specification is silent but a reference implementation is the de
facto definition.** Mark the check `basis: de-facto`, name the reference,
and it is settled the same way.
3. **Genuinely ambiguous** — the specification is silent and there is no
reference implementation to defer to. This is the only case that reaches
the rest of this document.

A check in the third case ships as `match: present`: it records what each
store did without judging it, because there is no rule yet to judge it
against. This file is how one gets a rule, so the check can be promoted to
`match: exact`.

## The process

1. **The disagreement is real, not a harness bug.** Before anything else,
rule out the alternative `CLAUDE.md` names: a poll shorter than a store's
commit interval, a read-back selector that assumed a label, a case
measured before a store finished starting. If ruling it out is not
possible from the case's own run, it is not ready for adjudication.
2. **Open an issue from the `Adjudication` template**
(`.github/ISSUE_TEMPLATE/adjudication.md`), naming the check, what every
implementation that has run it did, and precisely what the specification
does not settle — quoting the clause and saying where it stops, not
paraphrasing it.
3. **Put the question to the specification's owners** — the standards body,
or the project that maintains the de facto definition when there is no
formal spec — and link that discussion from the issue. An issue with no
upstream question linked stays open with nothing to promote: recording a
divergence is the point, and guessing at what the specification's authors
would say is exactly what this process exists to avoid.
4. **Record the answer verbatim in the issue and in the check's `notes:`**
when it arrives, with the date and who gave it.
5. **Promote the check only once a rule exists.** Change `match: present` to
`match: exact`, cite the answer in `rule:`, and re-run the suite: every
implementation that disagreed with the new rule is now `ALTER`, not
`present`, and that is the adjudication doing its job — a `present` check
never fails a backend for the divergence it records; an `exact` one does,
because there is now a rule it violates.

A check does not sit at `present` forever by default. Every one gets an
adjudication issue, even if the issue stays open for years waiting on an
answer — `AGENTS.md`'s "Never" section is explicit that a `present` case is
not simply left as the final answer.

## Two implementations disagreeing is not, on its own, a rule

The order of precedence above is deliberately not "whichever behaviour is
more common" or "whichever behaviour the newest implementation chose." A
specification that leaves something unsettled leaves it unsettled regardless
of how many stores converge on one answer by accident; convergence is
evidence worth citing in the issue, not a substitute for an answer from the
people who own the specification.

## Worked example

`cases/otlp-logs/timestamp-nanosecond-precision.yaml` is the first case this
process was written for. Five stores answered five different ways for how
much of `time_unix_nano`'s precision survives a round trip — millisecond
string, microsecond integer, a nanosecond integer that turned out to hold
only whole microseconds, and two that kept every digit. The OpenTelemetry
logs specification defines what the field means and says nothing about what a
receiver must retain, so no rule exists yet to promote the check with, no
matter how the five stores split. The case's own `notes:` record each
measurement as it was made; see there for the reproductions.
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ check exists to find.

Two implementations disagreeing is not a rule. A check moves from `present` to
`exact` only when a rule exists, not when the disagreement gets interesting.
[`ADJUDICATION.md`](ADJUDICATION.md) makes step 3 operational: the issue
template to use, what the upstream question has to establish, and what
"promote the check" actually changes.

### Do not tune a backend to fit the corpus

Expand Down
7 changes: 5 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,11 @@ value is entirely in every entry being defensible.

### If two backends disagree and you are unsure who is right

Say so in the pull request and leave the verdict out. Open an issue upstream, or
with whoever owns the specification, and let it be settled there.
Say so in the pull request and leave the verdict out — ship the check as
`match: present`. Then open an issue from the `Adjudication` template and work
through [`ADJUDICATION.md`](ADJUDICATION.md#the-process): the question goes to
whoever owns the specification, and the check is promoted to `match: exact`
only once they answer.

A wrong verdict published under this project's name damages someone's reputation
unfairly and destroys the project's own. Being slow is much cheaper than being
Expand Down
63 changes: 63 additions & 0 deletions GOVERNANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Governance

The results are only worth reading if they are not for sale. This is written
down before anyone has a reason to ask.

## Maintainers

The project currently has one maintainer, [@DeviousCardi](https://github.com/DeviousCardi),
who has commit access and decides what merges. That is a starting point, not
a target: a project with one maintainer has a bus factor of one, and this file
exists so growing past that happens by a stated process rather than by
whoever shows up first getting write access.

### How a maintainer is added

1. Several substantive pull requests merged — a check with a real citation, an
adapter re-confirmed by hand, or a runner fix, not typo corrections.
2. A public proposal, as an issue on this repository, naming the person and
what they have contributed.
3. No standing maintainer objects within two weeks.

Removal follows the same shape: a public issue, a reason, two weeks for
objection. A maintainer who has been inactive for two consecutive quarterly
reruns (`docs/BACKENDS.md`'s checklist, run by `.github/workflows/matrix.yml`)
is asked directly before being proposed for removal, not removed by default —
absence is not the same as disagreement.

## Vendor conflict of interest

A maintainer employed by, or paid by, a vendor whose backend appears in the
matrix **declares it in this file**, in the table below. A maintainer with a
declared interest in a vendor:

- Does not adjudicate a case (`ADJUDICATION.md`) where that vendor's backend
is the reference implementation or the backend under dispute.
- Does not merge a pull request that changes that vendor's adapter, files a
finding against it, or resolves a `divergence` issue about it — review and
approval come from another maintainer.
- May still write cases, fix the runner, and do everything else a maintainer
does; the recusal is scoped to decisions about the vendor they are
affiliated with, not a blanket exclusion.

An affiliation is declared before the maintainer takes any of the actions
above, not after being asked about it.

| Maintainer | Vendor | Relationship |
| --- | --- | --- |
| _(none declared)_ | | |

## Funding

None. If that changes, the amount and source are added to
[`README.md`](README.md#funding), not only here — a reader deciding whether
to trust a result should not have to find this file to learn who paid for it.

## Scope of this document

This covers who decides what merges and who may declare a vendor conflict of
interest away from a decision. It does not cover the rule for what a check
may assert, which is `AGENTS.md`'s "Adjudication" section and made
operational in [`ADJUDICATION.md`](ADJUDICATION.md); it does not cover how a
backend maintainer disputes a specific published verdict, which is
`.github/ISSUE_TEMPLATE/divergence.md`.
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ static `matrix.html` under `results/<date>/<suite>/`.
| `src/` | The runner. Sends cases, reads back, compares, decides verdicts. |
| `tools/` | The corpus gate CI runs. |
| [`AGENTS.md`](AGENTS.md) | The design, and how to add a check or a backend. |
| [`GOVERNANCE.md`](GOVERNANCE.md) | Who maintains this project, and vendor conflicts of interest. |
| [`ADJUDICATION.md`](ADJUDICATION.md) | How a `present` check earns a rule and becomes `exact`. |

The corpus is the asset. The runner is plumbing and could be rewritten in a
weekend; a corpus of checks that each trace to a line of a specification takes
Expand Down Expand Up @@ -200,7 +202,14 @@ The results are only worth reading if they are not for sale.
contribute to.

A comparison written by an interested party is an advertisement. This should not
become one.
become one. [`GOVERNANCE.md`](GOVERNANCE.md) has who maintains this project and
the rule for a maintainer with a vendor conflict of interest;
[`ADJUDICATION.md`](ADJUDICATION.md) has how a check earns the right to fail a
backend rather than only record what it did.

### Funding

None. If that changes, the amount and source are listed here.

## Licence

Expand Down
6 changes: 6 additions & 0 deletions cases/otlp-logs/timestamp-nanosecond-precision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -110,3 +110,9 @@ notes: >
is not the same as a rule saying it must be kept, and the specification still
does not give one. Recorded, not adjudicated — but this is now a strong
question to put to the specification's owners rather than a curiosity.

Adjudication opened 2026-09-09:
https://github.com/DeviousCardi/specmatrix/issues/12, per ADJUDICATION.md.
Open, waiting on the upstream question being put to the OpenTelemetry
specification's owners — this check stays `present` until that answer
arrives, whichever way it goes.
Loading