Skip to content

fix(evals): replace named organizations with generic descriptors - #3853

Closed
caio-pizzol wants to merge 1 commit into
mainfrom
fix/remove-customer-names-from-evals
Closed

fix(evals): replace named organizations with generic descriptors#3853
caio-pizzol wants to merge 1 commit into
mainfrom
fix/remove-customer-names-from-evals

Conversation

@caio-pizzol

@caio-pizzol caio-pizzol commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

The evals suite describes its scenarios by naming specific organizations and mapping each one to the workflow it uses. Those mappings are not needed to explain what the scenarios cover, and naming them here makes the relationships public.

This replaces each name with a description of the document workflow the scenario exercises. One test description elsewhere named the same organization as the source of a fixture, so it now refers to the reported fixture instead.

  • Task strings and their matching assertions were anonymized together, so the exact inputs and expected strings differ. The workflow each scenario exercises, and the coverage, stay equivalent.
  • Five files: the evals suite and README, one prompt example, and two test descriptions.
  • The two test-description edits are text only and touch no assertions.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

The evals suite described its scenarios by naming specific organizations and
mapping each to the workflow it uses. Those mappings are not needed to explain
what the scenarios cover, and naming them here makes the relationships public.

Describe each scenario by the document workflow it exercises instead. One test
description elsewhere named the same organization as the source of a fixture;
it now refers to the reported fixture. The scenarios, assertions, and coverage
are unchanged.
@caio-pizzol
caio-pizzol force-pushed the fix/remove-customer-names-from-evals branch from db7c224 to 8a3fce0 Compare July 28, 2026 00:40
@caio-pizzol
caio-pizzol marked this pull request as ready for review July 28, 2026 00:41
@caio-pizzol
caio-pizzol requested a review from a team as a code owner July 28, 2026 00:41
Comment thread evals/suites/execution/tests/customer-workflows.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants