Skip to content

Real workload: lazy TF imports + real-component service driver - #15

Open
andre-merzky wants to merge 25 commits into
feature/dtaas-twinfrom
feature/dtaas-twin-real
Open

Real workload: lazy TF imports + real-component service driver#15
andre-merzky wants to merge 25 commits into
feature/dtaas-twinfrom
feature/dtaas-twin-real

Conversation

@andre-merzky

@andre-merzky andre-merzky commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

The real-workload step for the service twin, stacked on the deploy-kit branch.

Lazy heavy imports so the real investigators package/instantiate on the client and broker without TensorFlow — it loads only in the task bodies, which run on the endpoint:

  • the three surrogate wrappers (fno/pinn/pcr) defer from .main import … into their task bodies;
  • each do_*/__init__.py swaps eager from .main import * for a PEP 562 __getattr__, so importing a submodule no longer drags TF/scikit-learn in, while tasks.<pkg>.tk_* still resolves on access (the wrapper.py executable path, which runs on the endpoint where TF lives).

Verified locally: the three investigators + wind_agent + sink + profiler all import TF-free; the only remaining client/broker deps are the pyspot submodule and python-dotenv (both light).

twin_service_real.py mirrors twin.py's graph with the real FNO/PINN/PCR investigators, real profiler + Pi predictor, real sink, and the external fake sensor. It's marked experimental — the starting point for the on-Perlmutter test cycle, not yet validated end to end.

Provisioning: setup-hpc-endpoint-real.sh (prior branch) clones Ben's cfdaai env and installs our runtime; this commit adds submodule init. Two open items documented in service/README.md: the shared-filesystem topology (run the broker on Perlmutter too for the real workload) and cloudpickle parity between the conda clone and ve.demo.

No behavior change to the faked demo or the standalone twin.py path (the __getattr__ preserves wrapper.py's contract).

🤖 Generated with Claude Code

https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU

andre-merzky and others added 2 commits September 4, 2026 11:00
@andre-merzky @claude
real workload: lazy heavy imports + a real-component service driver
bbddf04 
Lazy imports so the real investigators package/instantiate on the
client and broker without TensorFlow (it loads only in the task bodies,
which run on the endpoint):
  - the three surrogate wrappers (fno/pinn/pcr) defer `from .main
    import ...` into their task bodies;
  - each do_*/__init__.py replaces eager `from .main import *` with a
    PEP 562 __getattr__, so importing a submodule no longer drags TF /
    scikit-learn in, while `tasks.<pkg>.tk_*` still resolves on access
    (wrapper.py's executable path, on the endpoint).
Verified: the three investigators + wind_agent + sink + profiler all
import TF-free (pyspot submodule + dotenv are the only remaining
client/broker deps, both light).

twin_service_real.py mirrors twin.py's graph with the real FNO/PINN/PCR
investigators, real profiler + Pi predictor, real sink, and the
external fake sensor -- experimental, the on-Perlmutter starting point.
setup-hpc-endpoint-real.sh inits the pyspot submodule; README documents
the client deps and the two open items (shared-filesystem topology,
cloudpickle parity).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
profiler: append the Pi data.csv row on the endpoint, not the broker
73eaab5 
EndpointInvestigator.main_loop runs on the broker, but it appended new
rows to data.csv that train_model (an executable task) reads on the
endpoint -- different filesystems once broker and endpoint are split.
The append moves into a function_task (whose body runs on the endpoint),
so data.csv is produced and consumed on the same host.

This is the one cross-host file hand-off that is fixable without
staging.  The rest of the profiler chain uses executable_tasks, whose
bodies run on the engine (broker) to build the command while the command
runs on the endpoint -- so the .pkl / .json inputs they write are
broker-side and the commands read them endpoint-side.  Bridging that is
the ORBIT staging work (separate), not a filesystem assumption to fix
here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky
andre-merzky force-pushed the feature/dtaas-twin-real branch from f549dc7 to 73eaab5 Compare September 4, 2026 09:01
andre-merzky and others added 23 commits September 4, 2026 11:29
@andre-merzky @claude
profiler: measure the inference in-process (approach 1), not via subp…
1f85c06 
…rocess

The profiler cloudpickled the inference to a file and ran the standalone
profiler.py under a fresh process group.  Because an executable task's
command-builder runs on the broker while its command runs on the
endpoint, that .pkl was written broker-side and read endpoint-side --
a cross-host hand-off needing a shared filesystem.

inproc.py runs the call in-process on the endpoint and reads the
process's own counters (perf_counter, getrusage, psutil.io_counters),
returning the profiler's exact six-key schema so data.csv and
endpoint_trainer.py are unchanged.  ProfilerInvestigator and
EndpointInvestigator now profile via a function-task (endpoint-side, no
file, no subprocess).  Verified locally: real wall/cpu/mem on a dummy
inference; components import clean.

Trade-off (no process-group isolation, peak RSS not child PSS) is fine
for a comparative fingerprint.  PLAN-telemetry-profiling.md captures
approach 2 -- read the endpoint's own per-task telemetry instead of
re-running -- for later, plus the remaining executable-task input
hand-offs in the Pi-predictor (train/eval) that want the same treatment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real workload: stage inf.json endpoint-side; sink surfaces its heatmap
3936706 
- EndpointInvestigator.call_inference wrote inf.json in its
  executable-task command-builder (broker) while endpoint_eval.py reads
  it on the endpoint.  A stage_inf function-task now writes it on the
  endpoint before the command runs -- the last cross-host file hand-off
  in the profiler chain.  The datastore makedirs in __init__ (broker) is
  guarded; the endpoint tasks create it.
- CUPS_Sink renders the heatmap to bytes and calls record_output so the
  real driver shows heatmaps in the dashboard Outputs panel (needs
  digital.twins#38), and imports matplotlib lazily so packaging on the
  client/broker stays light.

Verified locally: all real-driver components import clean, TF-free and
matplotlib-free at import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real demo config: point the endpoint tasks tree and PLAYGROUND_DIR at us
36d7369 
- setup-hpc-endpoint-real.sh checks out feature/dtaas-twin-real for the
  endpoint tasks tree -- the base branch lacks the in-process profiler,
  the inf.json staging fix and the output-emitting sink.
- PLAYGROUND_DIR -> /pscratch/sd/m/merzky/xgf_playground (a real PM
  scratch path; the driver reads it from config.sh on the client and
  ships it to the components, which use it endpoint-side).

Demo-specific values; to be generalised later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: force-reinstall orbit on the endpoint; add client deps script
a0f0481 
- setup-hpc-endpoint-real.sh force-reinstalls radical.orbit: a cloned
  conda env may already carry a stale one that pip treats as satisfied,
  so the endpoint script (radical-orbit-endpoint.py) never installs.
- setup-client.sh installs the light client-side deps the real
  components import at packaging time (python-dotenv, numpy, pandas) and
  inits the pyspot submodule; TensorFlow/matplotlib stay lazy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: orbit log to scratch (HOME quota); backfill orbit deps
24ed0e4 
- run-hpc-endpoint.sh points RADICAL_ORBIT_LOG_FILE at scratch and sets
  the level to WARNING: the default ~/.radical/orbit/logs is on NERSC
  HOME, whose tiny quota the endpoint log (esp. at debug) blows out --
  the registration itself succeeds but every log write then errors.
- setup-hpc-endpoint-real.sh backfills orbit's own deps after the
  --no-deps script reinstall (fastapi/uvicorn/websockets), so a clone
  missing them still starts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: install rhapsody LAST, with extras -- fixes task_logs + opent…
212944d 
…elemetry

Two regressions from the conda-clone endpoint:
- pip install "$DT_DIR" (digitaltwin) ran after the rhapsody branch pin
  and pulled a different (main-based) rhapsody over it, reintroducing the
  Batch(task_logs=...) crash.  rhapsody now installs LAST so the
  dragon-compatible branch wins.
- the --no-deps rhapsody install skipped the telemetry extra's
  opentelemetry.  A second, extras install (no --no-deps) backfills
  opentelemetry + dragonhpc without re-resolving the pinned stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: fold every real-demo fix into the scripts (idempotent, site-s…
e110f44 
…pecific)

So a plain re-run of setup + run does everything, no one-off commands:
- setup-hpc-endpoint-real.sh seeds the Pi dataset at the RUNTIME datastore
  (PLAYGROUND_DIR/profiler/pi_profiler, where endpoint_trainer reads it),
  not just the tasks tree; PLAYGROUND_DIR is a script var matching config.sh.
- run-hpc-endpoint.sh clears any old HOME orbit log (NERSC quota) on top of
  redirecting the log to scratch at WARNING.
- setup-broker.sh backfills the telemetry extra's deps after the --no-deps
  rhapsody pin, mirroring the endpoint.

Paths are NERSC/demo-specific for now, to be generalised later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: consolidated one-command-per-role Sep_04 scripts
826a8cf 
Five self-contained scripts, each printing the demo banner and running
all-that's-needed for its role (deploy fix + env + data dirs + run):

  Sep_04_broker.sh          radical.3: install stack, pin dragon rhapsody, run broker
  Sep_04_endpoint_setup.sh  PM login node: checkouts, cfdaai clone, runtime, seed data
  Sep_04_endpoint.sh        PM allocation: env + launch endpoint under dragon
  Sep_04_sensor.sh          client: publish davis-wind
  Sep_04_client.sh          client: light deps + drive twin_service_real.py

Args are only the broker IP (and 'hpc' endpoint name where relevant);
everything else (SCRATCH paths, branch, PLAYGROUND_DIR, conda base env)
is baked in for the demo.  The endpoint is split because the allocation
cannot pip-install; setup runs on a login node first.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: banner shows only each script's own role
8b6c196 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service_real: ship whole tasks.* subtree by value
0211341 
add_agent failed on the broker with ModuleNotFoundError: no module
named 'utils_architecture'.  WindFieldAgent references tasks.davis (and
tasks.do_simulation, tasks.common.log_formatter), which import the
repo-root utils_architecture; the curated register_user_modules list
missed them, so they pickled by reference and the broker -- which has
no xGFabric checkout -- could not import them.  Sweep every already
imported tasks.* module plus utils_architecture instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: broker venv needs pandas (agent runs broker-side)
a6dbe2c 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
wind_agent: logger optional (service instantiates flow+config only)
2f90053 
The DTaaS service builds agents as cls(flow, *args, **engines); the
required 'logger' positional made add_agent raise TypeError.  Default
it to the module logger.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
profiler: invoke endpoint scripts as modules, not client __file__ path
531b44c 
script_path was os.path.realpath(__file__) captured at import on the
CLIENT; shipped by value, the executable/training commands then pointed
at the laptop checkout (/home/merzky/projects/xgfabric/...) while
running on Perlmutter.  Invoke via 'python3 -m tasks.profiler.endpoint_*'
instead -- the endpoint puts XGF_DIR on PYTHONPATH, so the path is
resolved on whatever host runs the command.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: endpoint env needs xgboost (+scikit-learn) for the Pi trainer
3d5fc36 
endpoint_trainer.py/endpoint_eval.py import xgboost; the cfdaai clone
does not carry it.  Install into the endpoint env at setup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real twin: add a visible learning lane on the wind stream
3f3aeb3 
The real learners (FNO/PINN/PCR, Pi predictor) run on the default
'inference' engine, so the dashboard showed no learning lane.  Add a
compact WindTrendLearner that consumes the same sensor stream and routes
its training to the 'learning' backend (the @function_task(backend=...)
label the service demo's SurrogateInvestigator uses).  It publishes to
its own WIND_TREND dtype, so it never competes with the WIND_FIELD path
feeding the sink.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/endpoint: expose CUDA driver so TF uses the GPU
4c5e034 
TF fell back to CPU because libcuda.so.1 was not on the task
LD_LIBRARY_PATH.  module load cudatoolkit + prepend /usr/lib64 (compute
node driver) and the env's CUDA libs; log nvidia-smi -L so we can see
whether the allocation actually has a GPU.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/client: default to the fast fake-surrogate driver on CPU
f6a4f3d 
No GPU allocation, so the real TF surrogates (~54s/inference on CPU) are
too slow for a live demo.  Default the demo client to twin_service.py --
fake SurrogateInvestigators (sleeps + numpy field synthesis, no TF or
xgboost) that produce heatmaps in seconds and carry the learning lane
(learn_backend='learning').  RUN_REAL=1 still selects twin_service_real.py
for a GPU-backed run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
service/sink: resolve heatmap file path endpoint-side, best-effort
c9c4e78 
main_loop runs on the broker; it built the PNG path from the broker's
home (XGF_WORKSPACE unset there -> ~/xgf_twin) and mkdir'd it there, then
handed that broker path to render_heatmap, which runs on the endpoint and
failed to write it (no such dir on Perlmutter).  Resolve XGF_WORKSPACE and
mkdir inside the task instead, and wrap the disk copy in try/except so it
can never fail the field -- the dashboard copy rides inline regardless.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/broker: force digitaltwin refresh (install.sh skips on version m…
1a338eb 
…atch)

install.sh installs digitaltwin by exact version string and skips when it
is unchanged, so the broker kept pre-record_output code and the sink
crashed with AttributeError: RuntimeAPI has no attribute record_output.
Force-reinstall the checked-out devel tip after install.sh.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/endpoint: gate GPU env behind RUN_GPU=1 (off by default)
4ce640e 
Prepending /usr/lib64 to LD_LIBRARY_PATH can shadow the conda env's libs
and break the dragon task launch -- the endpoint registers but its engine
never comes up, so session init times out.  Useless on the CPU/fake path
anyway; make it opt-in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service: drop the noisy per-poll state/calls line
7048fa8 
Keep the field-probe line as the heartbeat; the state dict is only used
for the failed-state check now.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service_real: refresh stale STATUS docstring
790d4d2 
Reflects reality after the demo cycle: runs end to end on the DTaaS
stack; the shared-FS caveat is resolved (compute + hand-offs are
endpoint-side); the TF path is GPU-bound so the demo defaults to the fake
driver (RUN_REAL=1 selects this one).  Points at the Sep_04 scripts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
service twin: telemetry reports from the endpoint's own recording
1837a3f 
Nothing to enable: the rhapsody plugin on the endpoint already records
every task plus a resource poll (the [telemetry] extra) into
telemetry-output/session.*.telemetry.jsonl.  collect_reports.py points
twin.py's existing report generators at those files -- waterfall,
dependency wait, stage timers, swimlane, concurrency/resources, and
the gantt render unmodified (verified against the service run's 8292
events).

Known gap, documented: per-workflow gantt grouping needs
asyncflow.workflow_id, which only workflow_scope() stamps -- an
engine-side telemetry feature for the DT service, tracked in
digital.twins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andre-merzky