Add bounded workflow label metrics dimensions

[jiaenren]

Summary

Issue - None

[OSMO-6501] Track B8 of the workflow-labels PR split.

• osmo_tasks_count gains one workflow_label_<key> attribute per curated policy key (any enforcement mode), value-clamped to the allow-list plus <missing> / <other> sentinels so cardinality stays bounded at |allow_list|+2 per key.
• Fixes the pre-existing counting defect: the query grouped by every column without COUNT(*), so each series read 1; it now selects and sums real counts.
• Label keys are escaped into Prometheus-safe attribute names (-, ., /, _ collision-free).
• The OSS workflow_resources_usage.json dashboard stays free of any deployment-specific label dimension, pinned by a new structural test; label-specific dashboards are provisioned per deployment.
• The osmo_label_validation_total{key,outcome} submit-gate counter landed with Enforce workflow label policy at submission #1223 (it is emitted inside the admission gate).

Stacked on #1225 for chain simplicity; functionally depends only on the schema (#1219) and policy config (#1220).
Extracted from prototype #1194.

Testing

• bazel test //src/service/core/workflow/tests:test_workflow_metrics (clamping matrix, sentinel values, attribute-name escaping, count aggregation, config-failure fallback)
• bazel test //src/service/core/workflow/tests:test_helpers
• bazel test //docs/deployment_guide/dashboards:test_dashboards

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added workflow-label dimensions to task metrics, including handling for missing, unexpected, and unsupported values.
  • Documented workflow metric labels, validation outcomes, and cardinality safeguards.
  • Added Grafana dashboard validation coverage for panel structure and deployment-neutral configuration.

• Bug Fixes

  • Improved recent-task aggregation to report grouped task counts with workflow labels.

• Tests

  • Added comprehensive coverage for workflow metrics, label policies, caching, disabled metrics, and dashboard configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1e843ce4-0b6a-47b5-a4cb-06d1bd536125

📥 Commits

Reviewing files that changed from the base of the PR and between b8fe273 and f3bcad8.

📒 Files selected for processing (8)

• docs/deployment_guide/dashboards/BUILD
• docs/deployment_guide/dashboards/test_dashboards.py
• docs/deployment_guide/references/configs_definitions/workflow.rst
• src/service/core/workflow/helpers.py
• src/service/core/workflow/tests/BUILD
• src/service/core/workflow/tests/test_helpers.py
• src/service/core/workflow/tests/test_workflow_metrics.py
• src/service/core/workflow/workflow_metrics.py

---

📝 Walkthrough

Walkthrough

Workflow task metrics now include curated workflow labels with bounded Prometheus attributes. Task queries return grouped counts. Tests cover aggregation, caching, disabled metrics, registration, and dashboard structure. Documentation describes label dimensions and validation metrics.

Changes

Workflow metrics

Layer / File(s)	Summary
Aggregated task-count query src/service/core/workflow/helpers.py, src/service/core/workflow/tests/test_helpers.py	get_recent_tasks selects workflow labels and COUNT(*), then groups by workflow UUID, status, and labels.
Curated workflow-label metrics src/service/core/workflow/workflow_metrics.py, src/service/core/workflow/tests/test_workflow_metrics.py, docs/deployment_guide/references/configs_definitions/workflow.rst, src/service/core/workflow/tests/BUILD	Workflow label policies are loaded and projected into bounded metric attributes. Tests cover fallback values, collisions, caching, disabled metrics, and registration metadata. Documentation defines the label dimensions and validation counter.
Dashboard and build validation docs/deployment_guide/dashboards/BUILD, docs/deployment_guide/dashboards/test_dashboards.py	A Bazel test target validates dashboard panel structure, variables, queries, and Kubernetes pod metrics.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• NVIDIA/OSMO#1224: Both changes modify workflow helpers and workflow-label handling.
• NVIDIA/OSMO#1225: This change adds metrics for workflow labels documented and configured by that PR.

Suggested reviewers: ryalinvidia, vvnpn-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 35.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding bounded workflow-label metric dimensions.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jiaenr/osmo-6501-b8-label-metrics

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📖 Docs preview: https://d3in15bfzp49i0.cloudfront.net/1226/index.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.91%. Comparing base (b8fe273) to head (f3bcad8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1226      +/-   ##
==========================================
+ Coverage   66.76%   66.91%   +0.14%     
==========================================
  Files         202      202              
  Lines       26050    26076      +26     
  Branches     3938     3942       +4     
==========================================
+ Hits        17393    17448      +55     
+ Misses       7917     7887      -30     
- Partials      740      741       +1     

Flag	Coverage Δ
backend	69.61% <100.00%> (+0.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/service/core/workflow/helpers.py	99.66% <ø> (ø)
src/service/core/workflow/workflow_metrics.py	90.90% <100.00%> (+61.49%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.