Skip to content

Add OBS-009..012 for agent-reliability-otel-labels label correctness - #184

Open
ericksondelacruz wants to merge 1 commit into
mainfrom
feat/otel-labels-correctness
Open

ericksondelacruz wants to merge 1 commit into
mainfrom
feat/otel-labels-correctness

Conversation

@ericksondelacruz

Copy link
Copy Markdown
Collaborator

Summary

This PR adds OBS-009..012 to observability/quality.yaml. OBS-006..008 check that the agent-reliability-otel-labels labels are written at all; these four repo-scope rules check that they are correct:

Rule Fires when Severity Confidence
OBS-009 A Run is shared between runs: built at module scope, in a class body, or as a Python default argument, or given a literal run id medium 0.6
OBS-010 Tool spans are labelled only on success low 0.55
OBS-011 Every tool-span label passes attempt as a constant low 0.7
OBS-012 No run-end label can record a failure low 0.55

Each rule is gated on observability being initialized and on the library being in use. Findings point at the offending call.

manifest.yaml raises schema_version from 22 to 23, because the rules use four new engine predicates and main already took 22 for LC-116.

Checks

  • scripts/check-rules-sync.sh in agent-reliability-analyzer, run against this branch: in sync, 128 files, 338 rules.
  • The rulebook gate run against this branch reports no OBS rule errors.

Merge order

  • Do not merge until agent-reliability-otel-labels 0.1.1 is published. The 0.1.0 README example trips OBS-009..012, so the first users flagged would be the ones who copied the official example.
  • Merge together with:
    • "Release 0.1.1: README examples that label failed tool calls and failed runs" (agent-reliability-otel-labels), which must also be published to PyPI and npm before this merges.
    • "Add OBS-009..012 for agent-reliability-otel-labels label correctness" (agent-reliability-analyzer).
    • "Rationale for OBS-009..012 (otel-labels label correctness) and LC-116" (trustabl-rulebook).
  • An engine build older than schema 23 loads this pack leniently: it skips OBS-009..012 and runs the rest.

… correctness

OBS-009 (shared Run, medium), OBS-010 (tool span labelled only on success,
low), OBS-011 (constant attempt, low) and OBS-012 (run end only on success,
low). New engine predicates raise schema_version from 22 to 23 (main took
22 for LC-116).

Do not merge before agent-reliability-otel-labels 0.1.1 is published: the
0.1.0 README example trips OBS-009..012.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants