[Pathways Elastic Training] Ensure one goodput monitoring flush at the end of the workload

[dipannita08]

Description

Elastic restart consists of both the config initialization and run. The final metrics flush (run once at the end of stop_goodput_uploader) is a one time operation that waits for all metrics to be computed and uploaded, however this is designed to be done at the actual end of the workload - not before an elastic event. This is where the elastic manager wraps the training loop. Doing this delays elastic restarts.

FIXES: b/484798225
FIXES: #123456

Tests

• E2E maxtext run, example: https://screenshot.googleplex.com/awT5LF4pL3beCNm
• Github CI

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[bvandermoon]

LGTM, thanks @dipannita08. Just to confirm my understanding, the final metrics flush will still happen at the end of elastic training, right?

[dipannita08]

LGTM, thanks @dipannita08. Just to confirm my understanding, the final metrics flush will still happen at the end of elastic training, right?

Yes it will still happen at the true end of the workload.