xGFabric twin as a service (DTaaS / ORBIT) - #13
Open
andre-merzky wants to merge 9 commits into
Open
Conversation
Same graph and the same selection story as twin.py -- Davis wind sensor, a field agent whose three surrogate architectures compete on profiler-predicted Pi runtime, a heatmap sink -- servicified: the twin lives in the ORBIT `dt` plugin, tasks run on a rhapsody endpoint, the sensor is an external channel publisher bound with `add_input`, the data plane is ORBIT. The service/* components ship by value and fake the physics at the seams twin.py already fakes (sensor records, tk_do_simulation); the real FNO/PINN/PCR training stacks are not imported. What stays real: the shared memoised simulation subtask across the three investigators, the profiler chain (an inline timed run of the shipped inference function), and pi-runtime-driven model selection. Verified end to end against a local broker + endpoint: twin running, per-window retraining visible in the probes, pcr (the cheapest architecture) winning selection, 28 heatmaps, 24 shared-sim cache hits, clean teardown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
Collaborator
|
Tested. Works great! Looks good to me to merge |
Collaborator
|
Had to comment out the saving of pictures on the service sink, as the path should be on Perlmutter rather than the broker's FS. (Would use the |
Collaborator
|
I like the dummy fake surrogate investigators and profiler model. Makes testing easy. Should migration to the real investigators and profiler be a separate PR? |
Same debugged constraints as the AmSC kit (dragon launcher, python >= 3.12.1, SLURM_EXPORT_ENV, PATH-by-name helpers, cert staging), with the xGFabric deltas: numpy/matplotlib for the by-value components, the endpoint registers as 'hpc' with the client's `remote` profile selecting it on dragon_v3, XGF_WORKSPACE on scratch -- and rhapsody pinned to fix/dragon-cancel-and-traceback on every tier, the branch carrying the idempotent-cancel fixes this demo surfaced (the orbit one runs broker-side). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
A home-quota casualty on the first remote attempt; the venv does not fit there. XGF_WORKSPACE already landed on scratch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The main-based fix/dragon-cancel-and-traceback pin broke the endpoint: main's dragon backend passes task_logs= to Batch(), which the pinned dragonhpc 0.14.1 does not accept (Session init failed on endpoint: Batch.__init__() got an unexpected keyword argument 'task_logs'). Both tiers now pin fix/dragon-cancel-idempotent (e491cd2-based) -- the branch the AmSC demo proved on this same dragon, carrying the dragon cancel + traceback fixes. Trade-off: the orbit-backend cancel fix lives only on the main-based #91 branch, so a harmless KeyError-cancel line may appear on broker teardown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The twin declared only an inference engine, so every task -- retraining included -- shared one dashboard lane. twin_service now declares a learning engine (same endpoint, concurrent executor by default; env knobs DT_LEARNING_ENDPOINT/_BACKEND), and each SurrogateInvestigator's train task carries backend="learning" so it routes there and shows as its own lane; inference, profiling and selection stay on the inference engine. learn_backend threads agent -> investigator and defaults to None (no label) so the components still work on a single-engine session. Verified locally: rhapsody.<sid>.inference and rhapsody.<sid>.learning both register, training routes to learning, selection still converges on pcr, no errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
…des) Level A provisioning for the actual FNO/PINN/PCR trainings, which need TensorFlow. Rather than build it, setup-hpc-endpoint-real.sh clones Ben's cfdaai conda env (which carries the stack) and installs our runtime into the CLONE -- never his shared env -- checks out the tasks tree the profiler shells into, stages the Pi-predictor dataset, and fails fast if the env is not Python 3.12 (wire contract). run-hpc-endpoint.sh now serves both modes: DT_VENV selects the venv (default ve.demo for the faked demo, the conda clone for real), and XGF_DIR puts the tasks tree on PYTHONPATH. Broker and client are unchanged -- they stay TF-free. README documents the path and the two gating items still open: lazy TensorFlow imports in the investigator wrappers (so packaging on client/broker does not need TF) and cloudpickle parity between the conda clone and ve.demo. Those are code changes on the investigators, tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The sink renders on the endpoint and returns the PNG bytes inline (a downscaled ~16 KB image, well under the return-value cap -- the DT service has no file staging), then records it via runtime.record_output so it shows in the dashboard's Outputs panel as fields arrive. Still written to XGF_WORKSPACE on the endpoint too. Needs digitaltwin's record_output (digital.twins#38). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
twin_service.pyruns twin.py's graph on a DTaaS broker for the Friday xGFabric call: twin in the ORBITdtplugin, tasks on a rhapsody endpoint, sensor as an external channel publisher (add_inputbinding), ORBIT data plane, client-side lifecycle/probe feedback loop.The
service/package holds service-safe components that ship by value: fake physics at the seams twin.py already fakes (sensor records,tk_do_simulation), while the demo's substance stays real — the shared memoised simulation subtask across the three investigators, the profiler chain (inline timed run of the shipped inference function instead of the exported-pickle subprocess, whose script path does not survive shipping by value), and model selection by profiler-predicted Pi runtime. The real FNO/PINN/PCR training stacks intasks/are untouched and not imported.Validated end to end against a local broker+endpoint:
pcr(cheapest architecture) wins selection, per-window retraining visible in the client probes, 28 sink heatmaps, 24 shared-sim cache hits, clean teardown, no framework errors.service/README.mdcarries the run recipe and the standalone→service mapping table.Found along the way, fix follows on the open rhapsody PR #91:
OrbitExecutionBackend.cancel_taskis not idempotent (teardown cancels race completed tasks intoKeyErrornoise) — the same defect class just fixed in the dragon backend.Telemetry is deliberately out of scope here; a follow-up branch attempts it.
🤖 Generated with Claude Code
https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU