Skip to content

[BENCH-835] Fix STT/TTS normalized validator false negatives - #606

Merged
kdmelon merged 1 commit into
mainfrom
dakota/bench-835-fix-stttts-validator-false-negatives-before-normalized
Sep 8, 2026
Merged

[BENCH-835] Fix STT/TTS normalized validator false negatives#606
kdmelon merged 1 commit into
mainfrom
dakota/bench-835-fix-stttts-validator-false-negatives-before-normalized

Conversation

@kdmelon

@kdmelon kdmelon commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Failed TTS requests with no audio filename and no TTFA measurement now reconcile only when an exact live-written provider-failure observation exists, including the canonical null timing artifact and its evaluation input. These cases remain ineligible for backfill.

STT observations whose metrics all fail with distinct valid errors now reconcile as succeeded observations with failed evaluations. Shared provider errors and ambiguous or malformed records retain their existing handling. Rollup comparison and read cutover behavior are unchanged.

Closes BENCH-835.

Testing

  • Full tests/unit/test_normalized_storage_backfill.py: 94 passed, including PostgreSQL-backed cases.
  • Changed-file Ruff check/format, strict mypy, and git diff --check: passed.
  • Local PostgreSQL 16.10 smoke used the real RunWriter and dual_write, then ran the migration CLI twice with database writes disabled. Both failure shapes reconciled with zero mismatches, identical reports, and unchanged table snapshots. Temporary servers were stopped.

Deployment and production cutover validation remain separate rollout steps.

@kdmelon
kdmelon requested a review from a team as a code owner September 8, 2026 20:16
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 7f26c081-9e0a-4a78-a2b8-f58820ba350e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kdmelon
kdmelon added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit f1c8d67 Sep 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants