dt-complete as a service: demo fixes, client feedback, deploy kit - #6
Merged
Conversation
…adical-collaboration/amsc into usability/refactor_usecase_api
Clarified comment regarding buffered inputs in simulation task.
…adical-collaboration/amsc into usability/refactor_usecase_api
The demo is stream driven, so all component output lands on the service and the client was silent for its whole four-minute run -- a stuck twin looked identical to a working one. The wait loop now polls the twin (state, verb counts, metrics) every 10s and probes get_inference with a fresh mock observation, printing true vs predicted gamma. Verified against a local broker: state/calls lines plus converging predictions from the first probe on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
First remote run said so exactly: "Backend 'dragon' not found. Available: [... 'dragon_v1', 'dragon_v2', 'dragon_v3', ...]". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
The module ships by value and its tasks run on the remote endpoint; the module-global workspace path (from __file__ on the client) named a directory that does not exist on Perlmutter, the training task failed its first write, and active_learn died with DependencyFailureError. The base now resolves inside the function on the executing host -- M3DC1_WORKSPACE overrides (e.g. $SCRATCH), default is the host's home. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
RandomForestRegressor(n_jobs=-1) makes joblib create a stdlib ThreadPool, which Dragon's mpbridge monkeypatching sends through DragonPool.__init__ -- super(type, obj) TypeError, task failed, active_learn dead. n_jobs=1 uses joblib's sequential backend and never creates a pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
- ROSE window tasks carry backend='learning' (the pre-#25 label seam, installed before the decorators run); the driver declares a learning engine on the same endpoint with the concurrent executor -- its own dashboard lane, no second dragon runtime, and the training tasks stay clear of dragon's mp bridge - the investigator reports val_r2 through the runtime's duck-typed component metrics, which feeds the dashboard's convergence bar Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
- ROSE task labels ride the decorators natively (backend= is a plain decor kwarg); the _register_task wrapper is gone. learn_backend is a constructor knob, None drops the label. - engine placement comes from the environment (DT_INFERENCE_ENDPOINT/ _BACKEND, DT_LEARNING_ENDPOINT/_BACKEND) with the HPC demo defaults; a laptop run overrides four variables instead of editing code. - the run window is --runtime (default 240s). - deploy/: setup + run scripts for the three tiers (broker host, HPC endpoint under the dragon launcher, client env) and a README carrying the hard-won constraints (dragon launcher requirement, python >= 3.12.1, SLURM_EXPORT_ENV, joblib-under-dragon, rhapsody branch pin). Validated end to end against a local broker: val_r2 metric live, predictions tracking, clean shutdown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything the remote end-to-end run of the servicified demo (broker on radical.3, dragon endpoint on Perlmutter, client + sensors on a laptop) surfaced, consolidated. Stacked on #5 — its commits appear here until it merges; the effective diff is the commits from
service demo: the client terminal shows lifecycle and predictionsonward.get_inference, printing true vs predicted gamma. A stuck twin is visible in seconds. Run window is--runtime.dragon_v3; the task workspace resolves at task runtime on the executing host (M3DC1_WORKSPACE, default the host's home — a client-side__file__path does not exist on Perlmutter); sequential forest fit (n_jobs=1— joblib's ThreadPool dies under Dragon's multiprocessing bridge).backend="learning"natively via decorator kwargs (constructor knoblearn_backend, no monkeypatching), the driver declares a learning engine (same endpoint, concurrent executor — no second dragon runtime), and the investigator reportsval_r2through the runtime's duck-typed component metrics: learning lane + convergence bar both live in the dashboard.DT_INFERENCE_ENDPOINT/_BACKEND,DT_LEARNING_ENDPOINT/_BACKEND, HPC defaults; a laptop run overrides four variables.SLURM_EXPORT_ENV=ALL, joblib-under-dragon, rhapsody branch pin until dragon+orbit backends: idempotent cancels; dragon failure tracebacks radical-cybertools/rhapsody#91 merges).Validated end to end locally (val_r2 metric live, predictions tracking, clean shutdown, zero broker errors) and remotely during the demo.
🤖 Generated with Claude Code
https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU