Skip to content

Improve training telemetry guards - #51

Merged
vikranthreddimasu merged 4 commits into
masterfrom
improve-training-telemetry-guards
Jul 29, 2026
Merged

vikranthreddimasu merged 4 commits into
masterfrom
improve-training-telemetry-guards

Conversation

@vikranthreddimasu

Copy link
Copy Markdown
Owner

Summary

  • Validate throughput loss and timestamp metrics before they enter tracker history.
  • Require loss recording inside a step context and only once per step.
  • Fix dataset-size remediation so tiny datasets never suggest impossible batch_size=0.
  • Keep optional YAML config import mypy-clean without making PyYAML mandatory.

Security changes

  • No authentication, authorization, credential, or secret-handling behavior changed.
  • Diff-only secret scan found no matches.

Usability changes

  • Training preflight errors now give valid remediation for impossible min_batches cases.
  • Telemetry failures now fail fast with direct ValueError/RuntimeError messages instead of poisoning dashboard/scheduler state.

Tests/checks run

  • python -m pytest tests/test_monitoring/test_throughput.py -q
  • python -m pytest tests/test_training/test_guards.py -q
  • make lint
  • make test outside sandbox: 1010 passed, 3 skipped, 2 warnings
  • git diff --check master...HEAD
  • git diff master...HEAD -- macfleet/ tests/ | rg secret-pattern scan: no matches

Commit list

  • e1cab1a Validate throughput loss metrics
  • cf692ae Fix dataset guard remediation advice
  • 49da166 Guard throughput loss recording lifecycle
  • 3a42601 Keep optional YAML import mypy-clean

Risks or manual review notes

  • Low risk: changes are validation/error-message focused in telemetry, training guards, and optional YAML typing.
  • Initial sandboxed make test failed because localhost bind was blocked; rerun outside sandbox passed.

Whether the PR was merged

  • Not merged yet; waiting for GitHub checks.

@vikranthreddimasu
vikranthreddimasu merged commit aa44ef7 into master Jul 29, 2026
11 checks passed
@vikranthreddimasu
vikranthreddimasu deleted the improve-training-telemetry-guards branch July 29, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant