Skip to content

xGFabric twin as a service: real workload + demo kit (supersedes #13, #14) - #16

Open
andre-merzky wants to merge 34 commits into
feature/dt-frameworkfrom
feature/dtaas-twin-real
Open

xGFabric twin as a service: real workload + demo kit (supersedes #13, #14)#16
andre-merzky wants to merge 34 commits into
feature/dt-frameworkfrom
feature/dtaas-twin-real

Conversation

@andre-merzky

@andre-merzky andre-merzky commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Supersedes #13 (fake DTaaS twin) and #14 (telemetry reports) — both are fully contained here (#13 as ancestor, #14's collect_reports.py + README cherry-picked in), so they can be closed in favour of this PR.

What's here

The xGFabric wind-field twin running as a DTaaS service on an ORBIT broker, with tasks on a rhapsody/dragon HPC endpoint. Two drivers share the same graph and components:

  • twin_service.py — fast fake-surrogate driver (sleeps + numpy field synthesis, no TF/xgboost). CPU-friendly, heatmaps in seconds; this is what the September demo runs.
  • twin_service_real.py — the real workload (FNO/PINN/PCR surrogates, real profiler + Pi predictor). GPU-bound, so it's opt-in via RUN_REAL=1.

Shared pieces: streaming learning lane (retraining on its own learning engine), profiler-predicted edge-runtime model selection, a heatmap sink that surfaces frames inline to the dashboard's Outputs panel, and the telemetry-report tool (service/collect_reports.py).

Demo kit

demo/Sep_04_{broker,endpoint_setup,endpoint,sensor,client}.sh — one command per role (arg: broker IP, plus hpc for the endpoint). Everything else (SCRATCH paths, conda base env, PLAYGROUND_DIR, branch pins) is baked in — deliberately site/demo-specific for now.

Notable fixes folded in

  • ship the whole tasks.* subtree by value (broker has no xGFabric checkout)
  • broker-side vs endpoint-side split resolved: data.csv append / inf.json staging are function_tasks; profiler measures in-process; no shared-FS requirement
  • endpoint scripts invoked as modules (-m), not client __file__ paths
  • sink resolves its output path endpoint-side, best-effort
  • RUN_GPU=1 gate for the CUDA env (off by default; it can break the dragon launch on CPU)

Not covered

Real TF timings need a GPU endpoint. rhapsody is pinned to the fix/dragon-cancel-idempotent branch (proper fix is rhapsody #91).

🤖 Generated with Claude Code

https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU

andre-merzky and others added 30 commits September 1, 2026 23:02
@andre-merzky @claude
xGFabric twin as a service: twin.py's graph on a DTaaS broker
2a6df55 
Same graph and the same selection story as twin.py -- Davis wind
sensor, a field agent whose three surrogate architectures compete on
profiler-predicted Pi runtime, a heatmap sink -- servicified: the twin
lives in the ORBIT `dt` plugin, tasks run on a rhapsody endpoint, the
sensor is an external channel publisher bound with `add_input`, the
data plane is ORBIT.

The service/* components ship by value and fake the physics at the
seams twin.py already fakes (sensor records, tk_do_simulation); the
real FNO/PINN/PCR training stacks are not imported.  What stays real:
the shared memoised simulation subtask across the three investigators,
the profiler chain (an inline timed run of the shipped inference
function), and pi-runtime-driven model selection.

Verified end to end against a local broker + endpoint: twin running,
per-window retraining visible in the probes, pcr (the cheapest
architecture) winning selection, 28 heatmaps, 24 shared-sim cache
hits, clean teardown.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@BenCarter44
Merge branch 'feature/dt-framework' into feature/dtaas-twin
b24b6a1
@BenCarter44
Comment out picture save, as requires endpoint filesystem
108b519
@andre-merzky @claude
service twin: r3/Perlmutter deploy kit, adapted from dt-complete
bccf99e 
Same debugged constraints as the AmSC kit (dragon launcher, python
>= 3.12.1, SLURM_EXPORT_ENV, PATH-by-name helpers, cert staging), with
the xGFabric deltas: numpy/matplotlib for the by-value components, the
endpoint registers as 'hpc' with the client's `remote` profile
selecting it on dragon_v3, XGF_WORKSPACE on scratch -- and rhapsody
pinned to fix/dragon-cancel-and-traceback on every tier, the branch
carrying the idempotent-cancel fixes this demo surfaced (the orbit one
runs broker-side).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: the Perlmutter checkout and venv default to $SCRATCH
09c8615 
A home-quota casualty on the first remote attempt; the venv does not
fit there.  XGF_WORKSPACE already landed on scratch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: pin the dragon-compatible rhapsody branch on both tiers
42f96db 
The main-based fix/dragon-cancel-and-traceback pin broke the endpoint:
main's dragon backend passes task_logs= to Batch(), which the pinned
dragonhpc 0.14.1 does not accept (Session init failed on endpoint:
Batch.__init__() got an unexpected keyword argument 'task_logs').  Both
tiers now pin fix/dragon-cancel-idempotent (e491cd2-based) -- the
branch the AmSC demo proved on this same dragon, carrying the dragon
cancel + traceback fixes.  Trade-off: the orbit-backend cancel fix
lives only on the main-based #91 branch, so a harmless KeyError-cancel
line may appear on broker teardown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
service twin: surrogate retraining runs on its own learning engine
709e54a 
The twin declared only an inference engine, so every task -- retraining
included -- shared one dashboard lane.  twin_service now declares a
learning engine (same endpoint, concurrent executor by default; env
knobs DT_LEARNING_ENDPOINT/_BACKEND), and each SurrogateInvestigator's
train task carries backend="learning" so it routes there and shows as
its own lane; inference, profiling and selection stay on the inference
engine.  learn_backend threads agent -> investigator and defaults to
None (no label) so the components still work on a single-engine session.

Verified locally: rhapsody.<sid>.inference and rhapsody.<sid>.learning
both register, training routes to learning, selection still converges
on pcr, no errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: real-workload endpoint kit (conda env, one run script, two mo…
651e1c5 
…des)

Level A provisioning for the actual FNO/PINN/PCR trainings, which need
TensorFlow.  Rather than build it, setup-hpc-endpoint-real.sh clones
Ben's cfdaai conda env (which carries the stack) and installs our
runtime into the CLONE -- never his shared env -- checks out the tasks
tree the profiler shells into, stages the Pi-predictor dataset, and
fails fast if the env is not Python 3.12 (wire contract).

run-hpc-endpoint.sh now serves both modes: DT_VENV selects the venv
(default ve.demo for the faked demo, the conda clone for real), and
XGF_DIR puts the tasks tree on PYTHONPATH.  Broker and client are
unchanged -- they stay TF-free.

README documents the path and the two gating items still open: lazy
TensorFlow imports in the investigator wrappers (so packaging on
client/broker does not need TF) and cloudpickle parity between the
conda clone and ve.demo.  Those are code changes on the investigators,
tracked separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
sink: render the heatmap to bytes and surface it to the dashboard
4c68c0e 
The sink renders on the endpoint and returns the PNG bytes inline (a
downscaled ~16 KB image, well under the return-value cap -- the DT
service has no file staging), then records it via
runtime.record_output so it shows in the dashboard's Outputs panel as
fields arrive.  Still written to XGF_WORKSPACE on the endpoint too.

Needs digitaltwin's record_output (digital.twins#38).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real workload: lazy heavy imports + a real-component service driver
bbddf04 
Lazy imports so the real investigators package/instantiate on the
client and broker without TensorFlow (it loads only in the task bodies,
which run on the endpoint):
  - the three surrogate wrappers (fno/pinn/pcr) defer `from .main
    import ...` into their task bodies;
  - each do_*/__init__.py replaces eager `from .main import *` with a
    PEP 562 __getattr__, so importing a submodule no longer drags TF /
    scikit-learn in, while `tasks.<pkg>.tk_*` still resolves on access
    (wrapper.py's executable path, on the endpoint).
Verified: the three investigators + wind_agent + sink + profiler all
import TF-free (pyspot submodule + dotenv are the only remaining
client/broker deps, both light).

twin_service_real.py mirrors twin.py's graph with the real FNO/PINN/PCR
investigators, real profiler + Pi predictor, real sink, and the
external fake sensor -- experimental, the on-Perlmutter starting point.
setup-hpc-endpoint-real.sh inits the pyspot submodule; README documents
the client deps and the two open items (shared-filesystem topology,
cloudpickle parity).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
profiler: append the Pi data.csv row on the endpoint, not the broker
73eaab5 
EndpointInvestigator.main_loop runs on the broker, but it appended new
rows to data.csv that train_model (an executable task) reads on the
endpoint -- different filesystems once broker and endpoint are split.
The append moves into a function_task (whose body runs on the endpoint),
so data.csv is produced and consumed on the same host.

This is the one cross-host file hand-off that is fixable without
staging.  The rest of the profiler chain uses executable_tasks, whose
bodies run on the engine (broker) to build the command while the command
runs on the endpoint -- so the .pkl / .json inputs they write are
broker-side and the commands read them endpoint-side.  Bridging that is
the ORBIT staging work (separate), not a filesystem assumption to fix
here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
profiler: measure the inference in-process (approach 1), not via subp…
1f85c06 
…rocess

The profiler cloudpickled the inference to a file and ran the standalone
profiler.py under a fresh process group.  Because an executable task's
command-builder runs on the broker while its command runs on the
endpoint, that .pkl was written broker-side and read endpoint-side --
a cross-host hand-off needing a shared filesystem.

inproc.py runs the call in-process on the endpoint and reads the
process's own counters (perf_counter, getrusage, psutil.io_counters),
returning the profiler's exact six-key schema so data.csv and
endpoint_trainer.py are unchanged.  ProfilerInvestigator and
EndpointInvestigator now profile via a function-task (endpoint-side, no
file, no subprocess).  Verified locally: real wall/cpu/mem on a dummy
inference; components import clean.

Trade-off (no process-group isolation, peak RSS not child PSS) is fine
for a comparative fingerprint.  PLAN-telemetry-profiling.md captures
approach 2 -- read the endpoint's own per-task telemetry instead of
re-running -- for later, plus the remaining executable-task input
hand-offs in the Pi-predictor (train/eval) that want the same treatment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real workload: stage inf.json endpoint-side; sink surfaces its heatmap
3936706 
- EndpointInvestigator.call_inference wrote inf.json in its
  executable-task command-builder (broker) while endpoint_eval.py reads
  it on the endpoint.  A stage_inf function-task now writes it on the
  endpoint before the command runs -- the last cross-host file hand-off
  in the profiler chain.  The datastore makedirs in __init__ (broker) is
  guarded; the endpoint tasks create it.
- CUPS_Sink renders the heatmap to bytes and calls record_output so the
  real driver shows heatmaps in the dashboard Outputs panel (needs
  digital.twins#38), and imports matplotlib lazily so packaging on the
  client/broker stays light.

Verified locally: all real-driver components import clean, TF-free and
matplotlib-free at import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real demo config: point the endpoint tasks tree and PLAYGROUND_DIR at us
36d7369 
- setup-hpc-endpoint-real.sh checks out feature/dtaas-twin-real for the
  endpoint tasks tree -- the base branch lacks the in-process profiler,
  the inf.json staging fix and the output-emitting sink.
- PLAYGROUND_DIR -> /pscratch/sd/m/merzky/xgf_playground (a real PM
  scratch path; the driver reads it from config.sh on the client and
  ships it to the components, which use it endpoint-side).

Demo-specific values; to be generalised later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: force-reinstall orbit on the endpoint; add client deps script
a0f0481 
- setup-hpc-endpoint-real.sh force-reinstalls radical.orbit: a cloned
  conda env may already carry a stale one that pip treats as satisfied,
  so the endpoint script (radical-orbit-endpoint.py) never installs.
- setup-client.sh installs the light client-side deps the real
  components import at packaging time (python-dotenv, numpy, pandas) and
  inits the pyspot submodule; TensorFlow/matplotlib stay lazy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: orbit log to scratch (HOME quota); backfill orbit deps
24ed0e4 
- run-hpc-endpoint.sh points RADICAL_ORBIT_LOG_FILE at scratch and sets
  the level to WARNING: the default ~/.radical/orbit/logs is on NERSC
  HOME, whose tiny quota the endpoint log (esp. at debug) blows out --
  the registration itself succeeds but every log write then errors.
- setup-hpc-endpoint-real.sh backfills orbit's own deps after the
  --no-deps script reinstall (fastapi/uvicorn/websockets), so a clone
  missing them still starts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: install rhapsody LAST, with extras -- fixes task_logs + opent…
212944d 
…elemetry

Two regressions from the conda-clone endpoint:
- pip install "$DT_DIR" (digitaltwin) ran after the rhapsody branch pin
  and pulled a different (main-based) rhapsody over it, reintroducing the
  Batch(task_logs=...) crash.  rhapsody now installs LAST so the
  dragon-compatible branch wins.
- the --no-deps rhapsody install skipped the telemetry extra's
  opentelemetry.  A second, extras install (no --no-deps) backfills
  opentelemetry + dragonhpc without re-resolving the pinned stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
deploy: fold every real-demo fix into the scripts (idempotent, site-s…
e110f44 
…pecific)

So a plain re-run of setup + run does everything, no one-off commands:
- setup-hpc-endpoint-real.sh seeds the Pi dataset at the RUNTIME datastore
  (PLAYGROUND_DIR/profiler/pi_profiler, where endpoint_trainer reads it),
  not just the tasks tree; PLAYGROUND_DIR is a script var matching config.sh.
- run-hpc-endpoint.sh clears any old HOME orbit log (NERSC quota) on top of
  redirecting the log to scratch at WARNING.
- setup-broker.sh backfills the telemetry extra's deps after the --no-deps
  rhapsody pin, mirroring the endpoint.

Paths are NERSC/demo-specific for now, to be generalised later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: consolidated one-command-per-role Sep_04 scripts
826a8cf 
Five self-contained scripts, each printing the demo banner and running
all-that's-needed for its role (deploy fix + env + data dirs + run):

  Sep_04_broker.sh          radical.3: install stack, pin dragon rhapsody, run broker
  Sep_04_endpoint_setup.sh  PM login node: checkouts, cfdaai clone, runtime, seed data
  Sep_04_endpoint.sh        PM allocation: env + launch endpoint under dragon
  Sep_04_sensor.sh          client: publish davis-wind
  Sep_04_client.sh          client: light deps + drive twin_service_real.py

Args are only the broker IP (and 'hpc' endpoint name where relevant);
everything else (SCRATCH paths, branch, PLAYGROUND_DIR, conda base env)
is baked in for the demo.  The endpoint is split because the allocation
cannot pip-install; setup runs on a login node first.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: banner shows only each script's own role
8b6c196 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service_real: ship whole tasks.* subtree by value
0211341 
add_agent failed on the broker with ModuleNotFoundError: no module
named 'utils_architecture'.  WindFieldAgent references tasks.davis (and
tasks.do_simulation, tasks.common.log_formatter), which import the
repo-root utils_architecture; the curated register_user_modules list
missed them, so they pickled by reference and the broker -- which has
no xGFabric checkout -- could not import them.  Sweep every already
imported tasks.* module plus utils_architecture instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: broker venv needs pandas (agent runs broker-side)
a6dbe2c 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
wind_agent: logger optional (service instantiates flow+config only)
2f90053 
The DTaaS service builds agents as cls(flow, *args, **engines); the
required 'logger' positional made add_agent raise TypeError.  Default
it to the module logger.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
profiler: invoke endpoint scripts as modules, not client __file__ path
531b44c 
script_path was os.path.realpath(__file__) captured at import on the
CLIENT; shipped by value, the executable/training commands then pointed
at the laptop checkout (/home/merzky/projects/xgfabric/...) while
running on Perlmutter.  Invoke via 'python3 -m tasks.profiler.endpoint_*'
instead -- the endpoint puts XGF_DIR on PYTHONPATH, so the path is
resolved on whatever host runs the command.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo: endpoint env needs xgboost (+scikit-learn) for the Pi trainer
3d5fc36 
endpoint_trainer.py/endpoint_eval.py import xgboost; the cfdaai clone
does not carry it.  Install into the endpoint env at setup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
real twin: add a visible learning lane on the wind stream
3f3aeb3 
The real learners (FNO/PINN/PCR, Pi predictor) run on the default
'inference' engine, so the dashboard showed no learning lane.  Add a
compact WindTrendLearner that consumes the same sensor stream and routes
its training to the 'learning' backend (the @function_task(backend=...)
label the service demo's SurrogateInvestigator uses).  It publishes to
its own WIND_TREND dtype, so it never competes with the WIND_FIELD path
feeding the sink.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/endpoint: expose CUDA driver so TF uses the GPU
4c5e034 
TF fell back to CPU because libcuda.so.1 was not on the task
LD_LIBRARY_PATH.  module load cudatoolkit + prepend /usr/lib64 (compute
node driver) and the env's CUDA libs; log nvidia-smi -L so we can see
whether the allocation actually has a GPU.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/client: default to the fast fake-surrogate driver on CPU
f6a4f3d 
No GPU allocation, so the real TF surrogates (~54s/inference on CPU) are
too slow for a live demo.  Default the demo client to twin_service.py --
fake SurrogateInvestigators (sleeps + numpy field synthesis, no TF or
xgboost) that produce heatmaps in seconds and carry the learning lane
(learn_backend='learning').  RUN_REAL=1 still selects twin_service_real.py
for a GPU-backed run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
service/sink: resolve heatmap file path endpoint-side, best-effort
c9c4e78 
main_loop runs on the broker; it built the PNG path from the broker's
home (XGF_WORKSPACE unset there -> ~/xgf_twin) and mkdir'd it there, then
handed that broker path to render_heatmap, which runs on the endpoint and
failed to write it (no such dir on Perlmutter).  Resolve XGF_WORKSPACE and
mkdir inside the task instead, and wrap the disk copy in try/except so it
can never fail the field -- the dashboard copy rides inline regardless.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
demo/broker: force digitaltwin refresh (install.sh skips on version m…
1a338eb 
…atch)

install.sh installs digitaltwin by exact version string and skips when it
is unchanged, so the broker kept pre-record_output code and the sink
crashed with AttributeError: RuntimeAPI has no attribute record_output.
Force-reinstall the checked-out devel tip after install.sh.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
andre-merzky and others added 4 commits September 4, 2026 17:47
@andre-merzky @claude
demo/endpoint: gate GPU env behind RUN_GPU=1 (off by default)
4ce640e 
Prepending /usr/lib64 to LD_LIBRARY_PATH can shadow the conda env's libs
and break the dragon task launch -- the endpoint registers but its engine
never comes up, so session init times out.  Useless on the CPU/fake path
anyway; make it opt-in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service: drop the noisy per-poll state/calls line
7048fa8 
Keep the field-probe line as the heartbeat; the state dict is only used
for the failed-state check now.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
twin_service_real: refresh stale STATUS docstring
790d4d2 
Reflects reality after the demo cycle: runs end to end on the DTaaS
stack; the shared-FS caveat is resolved (compute + hand-offs are
endpoint-side); the TF path is GPU-bound so the demo defaults to the fake
driver (RUN_REAL=1 selects this one).  Points at the Sep_04 scripts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
@andre-merzky @claude
service twin: telemetry reports from the endpoint's own recording
1837a3f 
Nothing to enable: the rhapsody plugin on the endpoint already records
every task plus a resource poll (the [telemetry] extra) into
telemetry-output/session.*.telemetry.jsonl.  collect_reports.py points
twin.py's existing report generators at those files -- waterfall,
dependency wait, stage timers, swimlane, concurrency/resources, and
the gantt render unmodified (verified against the service run's 8292
events).

Known gap, documented: per-workflow gantt grouping needs
asyncflow.workflow_id, which only workflow_scope() stamps -- an
engine-side telemetry feature for the DT service, tracked in
digital.twins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andre-merzky @BenCarter44