Rubric examples and counterexamples by pedro-pscunha · Pull Request #6 · pedro-pscunha/guideme-rust

pedro-pscunha · 2026-09-22T00:14:28Z

Implements the locked cross-SDK design for rubric examples in the Rust SDK, and releases it as
0.1.1. Python lands second, re-vendors spec/ and updates spec/SOURCE to this repository's
main after the squash-merge.

What this adds

Examples reach all three question kinds. Two repeatable #[guide(..)] keys, example = "…" and counterexample = "…", composing with
the existing rubric, key and fallback:

#[derive(Choice, Clone, Copy, PartialEq, Eq, Debug)]
enum Department {
    /// Payments, invoicing, refunds
    #[guide(example = "My card was charged twice", example = "Where is my refund?")]
    #[guide(counterexample = "The dashboard is down")]
    Billing,
    /// Bugs, outages, integrations
    #[guide(example = "502 on every request")]
    Technical,
    /// Pricing, upgrades, new accounts
    #[guide(fallback, example = "Do you have a team plan?")]
    Sales,
}

Billing reaches the wire as one string:

Payments, invoicing, refunds
Examples: My card was charged twice; Where is my refund?
Not this option: The dashboard is down

The proc macro renders at expansion time and emits a literal, so Options::RUBRIC and
Levels::LEVELS keep their types and question.rs, api/ and the wire are untouched. The
API's structured criteria objects are deliberately not used: a score answer echoes criteria
back in legend, which is BTreeMap<u8, String> here, so objects would mean a breaking change
to a public type in every SDK for a field none of them reads. Flattening measured as good and
~11% cheaper in billed input tokens. docs/design.md records the reasoning.

A noul has no enum to hang attributes off, so it takes Rubric:

noul("Is this ticket urgent?").criteria(
    Rubric::new("Something is broken now and nobody can work around it")
        .example("the checkout page is down"),
    Rubric::new("It can wait for the next working day")
        .example("a nightly job failed and we pull the numbers by hand for now"),
)

Question<Noul>::criteria already took impl Into<String>, so this is a new public type and
no signature change: a plain pair of strings keeps working. This is where examples earn the
most — see the live numbers below.

Rubric means the renderer now lives twice, at runtime in guideme and at expansion time in
the proc macro, because Options::RUBRIC is a const and cannot call a function. ~10 lines
duplicated, on purpose: sharing them costs a third published crate forever, and making RUBRIC
a function breaks a public trait. every_golden_row_renders_byte_for_byte_from_both_renderers
asserts a real derived enum's RUBRIC equals Rubric's output on every row, so the two cannot
drift silently.

The load-bearing invariant

A rubric with no examples and no counterexamples renders to itself, byte for byte. Every 0.1.0
declaration puts the same bytes on the wire. It is asserted three ways: a proptest over the
renderer in guideme-derive, the unchanged tests/derive.rs expectations, and a check on the
published spec/vectors/rubric.json. spec/schema/ and spec/vectors/policy.json regenerate
byte-identically, which is the evidence the wire contract did not move.

Contract

spec/vectors/rubric.json is new: the inputs and the rendering for each case, generated from
real derived enums in spec.rs rather than a second copy of the algorithm. docs/contract.md
gains a part 3 stating the algorithm, the invariant and the one deliberate asymmetry — Rust
renders in the proc macro, Python renders where a rubric becomes wire text — with equivalent
inputs producing identical wire bytes either way.

Validation

Each is a compile error naming the rule, with a trybuild case: an empty or whitespace-only
example or counterexample; a duplicate inside one variant's examples; an example on a
variant with no rubric to attach it to; counterexample on a Levels derive; the same string
as an example of two different variants; and the same string as both an example and a
counterexample of one variant. An example asserts that an input belongs here, so the last two
assert something that cannot be true.

This release breaks no existing build. Every new error needs an example or
counterexample to fire, and nothing in 0.1.0 could have written one. An empty or
whitespace-only rubric is rejected only where examples were attached to it, so
#[guide(rubric = "")] and a bare /// still compile exactly as they did — the same line
Python draws.

The same string as an example of one option and a counterexample of another stays legal and
is required — it is the confusable-options pattern the feature exists for, and tests/live.rs
uses it, which is what proves the carve-out holds.

Declaration order and the Not this option label are contract, not presentation. The vector
carries a case whose clauses are deliberately not in sorted order, so a renderer that sorted
would fail it.

A note on the test ceiling

AGENTS.md said "the ceiling is 30 nextest entries", which turned out to bound nothing
precisely: cargo nextest list counts the #[ignore]d live tests and cargo nextest run does
not, so the same tree reported 28 or 31 depending on which you meant. This PR needed the room,
and deciding what the rule means inside the change that benefits from it is exactly the wrong
order.

So it was not decided here. The line is redefined to bound the entries the gate runs —
what cargo nextest run --workspace reports, 29 today — with the #[ignore]d live tests
explicitly outside it, because they never execute in the gate and so are not what the ceiling
protects. The number 30 is unchanged, and the redefinition was ratified separately by the
repository owner rather than as part of this change. The reviewer's concern here was precedent,
not the merits: a project that quietly reinterprets its own limits when they bind has no
limits. The merits were never in question — nothing about a test that does not run protects the
gate's runtime.

Verification

mise run check green through the pre-push hook: fmt-check, clippy -D warnings, 27 nextest
entries (30 counting the #[ignore]d live tests, at the ceiling AGENTS.md sets), doctests,
rustdoc, cargo-deny.
Live against jev-1.13.0, #[ignore]d like the other live tests, asserting direction and
never a number. On the design note's ambiguous input
About those shoes - what is the situation with the money side of things?, probability on
a bare rubric picks return_policy — the wrong answer — at 0.78, and the same rubrics
carrying examples pick return_status at 0.88. The assert is the pick plus the direction.
On the noul case,
Our nightly export job has been failing since Tuesday. We pull the numbers by hand for now.
asked as "Is this ticket urgent?" reads 0.74 with plain criteria and 0.22 with criteria that
name a broken job with a manual workaround — and 0.22 is the right answer.
examples/otlp/Cargo.lock refreshed for the version bump; cargo build --locked in that
directory passes as CI runs it.

Callers want to attach inputs that belong to an option or a level, and inputs that do not, so the model has something concrete to match against. Measured against live Jev, the gain comes from the examples being present, not from sending the API's structured criteria objects: flattening scores the same, costs ~11% fewer input tokens, and leaves the wire types alone. `example` and `counterexample` are repeatable `#[guide(..)]` keys; the macro renders the final string at expansion time, so `Options::RUBRIC` and `Levels::LEVELS` keep their types and nothing below the derive changes. A variant with neither renders to its rubric itself, byte for byte, which is what keeps every existing request unchanged.

The rendered rubric string is now a cross-SDK contract item, so it needs an artifact the other SDKs can test against. `spec/vectors/rubric.json` carries the inputs and the rendering for each case, read back from real `#[derive(Choice)]` and `#[derive(Levels)]` enums: the vector is provably what the macro emits, not a second implementation of the algorithm. Declaring those enums inside the crate needs `extern crate self as guideme`, because the derives expand to `::guideme` paths. The schemas and the policy vectors regenerate byte for byte, which is the evidence that the wire is untouched.

Model output is not stable to two decimal places, so the opt-in live test asserts the invariant the feature exists for rather than a float: on the design note's confusable return_policy/return_status pair, naming the inputs that belong to return_status moves probability onto it (0.63 -> 0.71 against jev-1.13.0).

The composition is shared with every other SDK, so it belongs in docs/contract.md next to the wire schemas and the policy vectors, along with the one deliberate asymmetry: Rust renders in the proc macro, Python renders where a rubric becomes wire text. docs/design.md records why we flattened instead of sending the API's structured criteria, because the next person will ask why a documented API feature was ignored.

Purely additive, so a patch. `guideme-derive` is pinned by version as well as by path, so the pin moves with it: a `guideme` 0.1.1 that resolved `guideme-derive` 0.1.0 would not know the new attribute keys. `examples/otlp` pins the library through a path dependency and CI builds it `--locked`, so its lock file is refreshed here.

A noul's true/false criteria are as confusable as a choice's options, and measured the largest swing of the three kinds: on a ticket describing a broken job with a manual workaround, plain criteria answer 0.75 urgent and criteria carrying examples answer 0.25, which is the right answer. Leaving noul out was a real gap. A noul has no enum to hang attributes off, so the parts arrive as a value. A builder rather than a second criteria method because `criteria` already takes `impl Into<String>`: a Rubric drops in with no signature change, and a plain pair of strings keeps working.

An example asserts that an input belongs to this option, so the same string on two options asserts it belongs to both, and the same string as an example and a counterexample of one option asserts it is in and out. Neither can be true; both are now compile errors. The same string as an example of one option and a counterexample of another stays legal — that is the confusable-options pattern the feature exists for, and tests/live.rs relies on it. The golden rows deliberately reuse a string across cases, which no single declaration may now do, so the spec and test fixtures split across two enums. One gains a case whose clauses are not in sorted order: declaration order is contract, and nothing pinned it before. The passthrough property test moves to guideme, over `Rubric`, and the golden test pins the derive's output equal to it. That is the whole point of the duplicated renderer: one law, property-tested once, and the macro tied to it.

The choice case was already covered. The noul case is where examples swing hardest, so it is the one worth spending a live request on: against jev-1.13.0 the ticket reads 0.74 urgent with plain criteria and 0.22 with criteria that name a broken job with a manual workaround. The assert is the direction, never the number.

Three of the five changes are contract: all three question kinds take examples, rendering follows declaration order, and `Not this option` is the measured label rather than a taste call. The duplicated renderer and the deliberately unwidened choose_among/score_levels signatures are the two things the next person will want to "fix", so both say why in docs/design.md.

An empty rubric with no examples has nothing to do with this feature, and 0.1.0 compiled it. Rejecting it unconditionally made 0.1.1 break builds that never asked for anything new, and the Python SDK, reading the same rule, drew the line the other way — a declaration legal in one SDK and not the other is a contract defect whichever rule is better. The error now fires only on a variant carrying an example or a counterexample: you described nothing. That matches Python's `option("", examples=[…])` and restores the property this design rests on, that 0.1.1 changes nothing for anyone not using the feature. tests/derive.rs pins a blank rubric still compiling.

The wire test asserts the derive's bytes against the same hand-written constants for all eight rows, and proves they survive to the request, so the golden-row test restated it. What it alone still carried was the runtime-against-derive pin for shapes the noul row does not reach, so that loop moves across rather than going away. The spec drift check cannot take this job: its vector is generated from the derive, so it re-states whatever the renderer currently does. Only a table written by hand can say the renderer is wrong.

The old wording left the bare rubric already picking return_status, so the test could only assert a nudge. With rubrics measured to leave the question genuinely ambiguous, the bare form picks return_policy — wrong — and the guided form picks return_status: p(return_status) 0.22 -> 0.88 against jev-1.13.0. The assert is the pick plus the direction, still never a number.

Every new compile error needs an example or a counterexample to fire, and nothing in 0.1.0 could have written one. That is the guarantee worth saying out loud in a patch release, and it is only true now that the empty-rubric rule is narrowed. docs/contract.md says where the line falls, so the next SDK draws it in the same place.

There are two copies of the algorithm; pinning one against the other on eight fixed rows leaves the macro free to diverge on anything outside them — a string carrying a newline or a "; " of its own, say. The pin catches drift on the rows it lists; only the property catches it everywhere else, so both copies get it.

Rubric::new("").example(..) rendered a stray "Examples:" line onto nothing, while Python refuses the same thing when it is constructed. A declaration accepted by one SDK and refused by the other is a contract defect however it renders. A builder has no Result to return, so criteria widens from impl Into<String> to impl Into<Rubric> to get an error channel, and Noul::wire returns Error::Config for the blank-with-examples case only — the same line the derive draws, so a blank description on its own still goes through as it did in 0.1.0. Widening is source-compatible, and that is checked rather than asserted: Rubric converts from every type String converts from, and tests/rubric.rs compiles one call of each shape. &String, Cow<str>, Box<str> and char all broke with only the two conversions the amendment named.

Thirty what was ambiguous, and the two readings differ by the three #[ignore]d live tests that never run in the gate. The ceiling exists to keep the gate fast, so it bounds what the gate runs; saying so, with today's figure, stops the question recurring. The number is unchanged.

criteria took impl Into<String> in 0.1.0. Narrowing it to impl Into<Rubric> broke &String, Cow<str>, Box<str>, char and — unfixably by any set of From impls — a caller's own helper generic over Into<String>. A sealed IntoRubric with a blanket case for Into<String> and one more for Rubric covers all of them, so this is a widening rather than a swap. The price is that Rubric must not be Into<String>, or the two cases overlap, so it renders through Display. Rubric also enforced none of the validation docs/contract.md calls shared: an empty example, a duplicate within a clause, and a string that was both an example and a counterexample all rendered silently, and every one of those is visible inside a single rubric. They move into into_wire alongside the blank description check, in the derive's wording. Only the checks that compare two options need a whole declaration, and those stay in the derive. The renderer pin moves out of the wire test: it is the only thing stopping two copies of the algorithm drifting, and it was running after a mock server had to start. The compatibility claim becomes a const _ block of one call per shape, so deleting a conversion is a build failure rather than a silent narrowing.

AGENTS.md says every misuse gets a trybuild case; empty and duplicate counterexample had none. The property regex excluded control characters, so a rubric containing a newline — exactly what "used verbatim, never trimmed" is there to promise — was never generated.

Every case came from a derive, so the vector said nothing about the kind with the largest measured win and nothing about the runtime renderer. Two noul cases come straight from a Rubric, which covers both. The drift guard checked starts_with plus contains, which would pass a renderer that reordered the clauses or changed the separator while the error claimed it had verified the rendering. It now re-renders each case through Rubric and refuses to publish a vector the two renderers disagree on.

"Empty or whitespace-only" and "duplicate" both read as obvious and are not: Rust trims on Unicode White_Space, Python's str.strip() also takes the C0 separators, and duplicate detection is exact while the emptiness check trims, so "a" and " a" legitimately coexist. A third SDK would have guessed. The vector's kind field was undocumented too. Also corrects the reasoning for criteria: under IntoRubric the accepted set grows rather than shrinks, so "widened" is true again, and the round's general lesson goes in design.md — compatibility claims get compiled, not reasoned about.

Display rendered without checking anything, and it was the documented route into the &str constructors — so the path the docs recommend for untrusted rubric text skipped every rule the type claims to enforce. render() is fallible and is now the only way to get the string, so there is no unchecked path to the wire. Rubric is unreleased, so nothing depends on the old conversion. Noul::wire held both rubrics in one tuple and rendered them independently, so a yes and a no could share an example — an assertion that one input belongs to both sides, which Python already rejected. It is the one runtime path with two rubrics in view, so it is the one place the derive's cross-option rule can apply here. choose_among still has no such moment. Also records that Rubric can never gain From<Rubric> for String: the blanket IntoRubric impl covers every T: Into<String>, so adding it would make the two impls overlap. That is not a 0.2.0 tradeoff, it is closed.

with_capacity was still counting three grids after the noul cases landed, so the vector reallocated on every run.

…omise The kind field was made binding on both SDKs and never actually landed here; I reported it closed after a patch script exited before writing the file. Items are also inserted verbatim, with no escaping and no promise that the rendering can be read back, which a third implementer would otherwise have to guess at. Narrows the claim that cross-option rules are compile-time only in Rust: true for choose_among, false for a noul, which holds both rubrics at once. Moves the notes back under Unreleased, since nothing is published and the date had already gone stale.

Items are joined onto one line, so a line break in one renders as a clause boundary the declaration never wrote. What that buys is rendering integrity, not input trust: a string rendered from declared parts carries exactly the clauses those parts declared. A description may still contain anything — it renders verbatim and always did, and constraining it would refuse legitimate prose while protecting nobody. U+000D is in for hygiene rather than forgery: a lone carriage return is pasted text that breaks the joined line. U+2028, U+2029 and U+0085 are deliberately out, being whitespace already and harmless embedded. The check is `contains` over those two code points and never a line-splitting primitive, because `str::lines` and `str.splitlines` cover different sets and would leave the two SDKs disagreeing while both looked correct. The derive checks after the blank-rubric rule rather than in `push_item`, so a variant breaking both reports the blank one as `Rubric::render` does. `push_item` runs while attributes are still being read and cannot see a rubric set by a later one, so precedence is not expressible there.

The shared-example assertion this file was meant to gain last round never landed: rustfmt had rewrapped the anchor before the patch ran and the patch did not check that it matched, so it silently did nothing while the source change it covered went in. render_pair was live and untested. Adds it back alongside the line-break case and a precedence case proving blank wins, and folds the four rejections through one helper so each reads as a line.

One sentence replaces the per-case notes: it explains why "a; b" as one item renders like two and still passes the duplicate check, why " a" is not "a", and why "A" is not "a" — and it stops a third SDK inventing normalisation. States the refused-versus-documented line with its reason, so it does not read as arbitrary: "; " and a literal "Not this option: x" change what a reader sees inside a clause, a line break changes which clause they are in, and only the second forges a clause that was never declared.

…t have Both SDKs reject a counterexample on a level — Rust as a compile error on the Levels derive — but this document only said a level never renders one. A third SDK implementing from it could build silent dropping instead of rejection, and the difference would not show until someone compared outputs. The refusal joins the rejected-loudly list; the rendering sentence stays, since it explains why. That sentence also claimed more than is true: score_levels takes a pre-rendered string, so a rubric carrying a counterexample can reach a level that way and render the clause. The guarantee is over declared parts, and now says so. Python refuses an explicitly empty examples=[], which the attribute surface cannot express at all — a variant either carries the attribute or does not. A reader diffing the two lists found a rule in one and silence in the other, with nothing to separate "not applicable" from "forgotten".

pedro-pscunha added 27 commits September 21, 2026 21:08

fix(spec): size the rubric vector for the noul cases

70735c3

with_capacity was still counting three grids after the noul cases landed, so the vector reallocated on every run.

pedro-pscunha merged commit d908ecf into main Sep 22, 2026
4 checks passed

pedro-pscunha deleted the feat/rubric-examples branch September 22, 2026 11:04

pedro-pscunha mentioned this pull request Sep 22, 2026

Name the escape path under the level rule, and date 0.1.1 #7

Merged

Navigation Menu

Repository navigation

Rubric examples and counterexamples - #6

Rubric examples and counterexamples#6
pedro-pscunha merged 27 commits into
mainfrom
feat/rubric-examples

pedro-pscunha commented Sep 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pedro-pscunha commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

The load-bearing invariant

Contract

Validation

A note on the test ceiling

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pedro-pscunha commented Sep 22, 2026 •

edited

Loading