diff --git a/README.md b/README.md index 0a162185..f839e15f 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ These dockerfiles will also be pushed and actively maintained in their original | Vision | [`OpenCV`](packages/vision/opencv), [`SAM`](packages/vision/sam), [`MobileSAM`](packages/vision/mobilesam), [`ncnn`](packages/vision/ncnn), [`DINOv3`](packages/vision/dinov3), [`SAM3`](packages/vision/sam3), [`Ultralytics`](packages/vision/ultralytics) | | Ryzen AI NPU | [`XDNA`](packages/npu/xdna), [`IRON`](packages/npu/iron), [`NPUEval`](packages/npu/npueval), [`Ryzen AI CVML`](packages/npu/ryzenai_cvml) | | Adaptive SoCs | [`PYNQ.remote`](packages/adaptive-socs/pynq-remote) | +| Federated Learning | [`Flower`](packages/federated/flower-base) ([`SuperLink`](packages/federated/flower-superlink), [`SuperNode`](packages/federated/flower-supernode), [`SuperExec`](packages/federated/flower-superexec)) | | Utilities | [`JupyterLab`](packages/ide/jupyterlab), [`amdgpu_top`](packages/init/amdgpu_top) | --- diff --git a/packages/federated/flower-base/Dockerfile b/packages/federated/flower-base/Dockerfile new file mode 100644 index 00000000..71a663a7 --- /dev/null +++ b/packages/federated/flower-base/Dockerfile @@ -0,0 +1,37 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +ENV DEBIAN_FRONTEND=noninteractive + +# Flower CLI binaries (flower-superlink, flower-supernode, flower-superexec) +# all come from this single pip package. Torch + ROCm is provided by the +# default rocm/pytorch base image, so no torch install is needed here. +ARG FLWR_VERSION=1.30.0 +RUN pip3 install --no-cache-dir --break-system-packages \ + "flwr[simulation]==${FLWR_VERSION}" \ + "flwr-datasets[vision]>=0.5.0" + +# Python 3.14 fix. The base image is now py3.14, but flwr-datasets 0.5.0 +# hard-pins datasets<=3.1.0, which drags in dill 0.3.8. That old +# datasets/dill combo runs a fingerprinting path that trips over py3.14's +# changed pickle.Pickler._batch_setitems() signature: +# TypeError: _batch_setitems() takes 2 positional arguments but 3 were given +# raised the moment a run loads a dataset (e.g. the ServerApp's CIFAR-10 +# eval set), aborting the whole federation. datasets 4.x drops that legacy +# code path and works under py3.14. It must be installed in a SEPARATE step: +# resolving it together with flwr-datasets fails (its <=3.1.0 cap), and +# bumping flwr-datasets to 0.6.0 instead is blocked by a rich pin conflict +# with flwr 1.32.0. The residual pip "incompatible" notice about the 3.1.0 +# cap is cosmetic — 0.5.0 works with datasets 4.x at runtime. +ARG DATASETS_VERSION=4.8.5 +RUN pip3 install --no-cache-dir --break-system-packages \ + "datasets==${DATASETS_VERSION}" + +WORKDIR /ryzers +COPY test.sh /ryzers/test_flower-base.sh +RUN chmod +x /ryzers/test_flower-base.sh + +CMD /ryzers/test_flower-base.sh diff --git a/packages/federated/flower-base/README.md b/packages/federated/flower-base/README.md new file mode 100644 index 00000000..913afa99 --- /dev/null +++ b/packages/federated/flower-base/README.md @@ -0,0 +1,49 @@ +# Flower Base Docker Setup + +Shared base layer for the Flower federated-learning Ryzers. Installs the +[Flower](https://flower.ai) framework (`flwr[simulation]==1.26.1`) on top +of the default ROCm/PyTorch base image, so the three flower binaries +(`flower-superlink`, `flower-supernode`, `flower-superexec`) and a +ROCm-enabled PyTorch are available to layers that build on this one. + +You usually don't build or run this Ryzer on its own — chain it with one of +the role Ryzers: + +```sh +# --name sets the final image tag (it defaults to "ryzerdocker", not the +# last package name), so each role gets its own image + run-script. +ryzers build --name flower-superlink flower-base flower-superlink # server box +ryzers build --name flower-supernode flower-base flower-supernode # client box +ryzers build --name flower-superexec flower-base flower-superexec # serverapp or clientapp runner +``` + +## Build & Run (standalone smoke test) + +```sh +ryzers build flower-base +ryzers run +``` + +The default `CMD` runs `test_flower-base.sh`, which verifies the CLI +binaries are installed and that `torch.cuda.is_available()` returns true +under ROCm. + +## Local single-machine smoke test + +For an end-to-end test on one host (SuperLink + ServerApp + two +SuperNode/ClientApp pairs) run [`../run-local.sh`](../run-local.sh). It +builds the three role Ryzers, brings everything up under `--network host` +on the loopback (with ClientAppIo ports offset by partition ID), submits +the quickstart-pytorch run, and tears down on exit. + +The same three role Ryzers (`flower-superlink`, `flower-supernode`, +`flower-superexec`) are also what you run in a distributed deployment — +just set `SUPERLINK_IP` to the server's IP on the client boxes. See each +role Ryzer's README for the multi-machine flow. + +## References + +- [Flower documentation](https://flower.ai/docs/framework/) +- [Multi-machine Docker tutorial](https://flower.ai/docs/framework/docker/tutorial-deploy-on-multiple-machines.html) + +Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. diff --git a/packages/federated/flower-base/config.yaml b/packages/federated/flower-base/config.yaml new file mode 100644 index 00000000..979668fa --- /dev/null +++ b/packages/federated/flower-base/config.yaml @@ -0,0 +1,12 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +# flower-base is a composable layer. It provides the flwr CLI on top of the +# default ROCm/PyTorch base image. Most users will not run it directly; +# instead chain it with one of: +# ryzers build --name flower-superlink flower-base flower-superlink +# ryzers build --name flower-supernode flower-base flower-supernode +# ryzers build --name flower-superexec flower-base flower-superexec + +build_arguments: +- "FLWR_VERSION=1.32.0" diff --git a/packages/federated/flower-base/test.sh b/packages/federated/flower-base/test.sh new file mode 100644 index 00000000..0c01c022 --- /dev/null +++ b/packages/federated/flower-base/test.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +set -e + +echo "Running tests for flower-base..." + +# Verify all three flower CLI entrypoints are installed +for bin in flower-superlink flower-supernode flower-superexec flwr; do + if ! command -v "$bin" >/dev/null 2>&1; then + echo "FAIL: $bin not on PATH" + exit 1 + fi + echo "Found: $(command -v "$bin")" +done + +# Verify flwr Python package and torch+ROCm +python3 - <<'PY' +import flwr +import torch +print(f"flwr version: {flwr.__version__}") +print(f"torch version: {torch.__version__}") +print(f"torch.cuda.is_available(): {torch.cuda.is_available()}") +print(f"torch HIP version: {torch.version.hip}") +PY + +echo "Tests passed!" diff --git a/packages/federated/flower-superexec/Dockerfile b/packages/federated/flower-superexec/Dockerfile new file mode 100644 index 00000000..84764f1c --- /dev/null +++ b/packages/federated/flower-superexec/Dockerfile @@ -0,0 +1,38 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +# Install the example Flower app (quickstart-pytorch). The pyproject.toml +# shipped here has torch dependencies stripped so we don't overwrite the +# ROCm PyTorch provided by the base image. +WORKDIR /app +COPY app/ /app/ +RUN pip3 install --no-cache-dir --break-system-packages /app + +# Re-assert the py3.14-compatible datasets. `pip install /app` above +# re-resolves the app's deps and, because flwr-datasets 0.5.0 hard-pins +# datasets<=3.1.0, downgrades datasets from the 4.x pinned in flower-base +# back to 3.1.0 — which reintroduces the py3.14 dill/pickle crash +# (_batch_setitems() TypeError) on the first dataset load. Keep this in +# lockstep with DATASETS_VERSION in flower-base/Dockerfile. +ARG DATASETS_VERSION=4.8.5 +RUN pip3 install --no-cache-dir --break-system-packages \ + "datasets==${DATASETS_VERSION}" + +WORKDIR /ryzers +COPY test.sh /ryzers/test_flower-superexec.sh +COPY run-superexec.sh /ryzers/run-superexec.sh +RUN chmod +x /ryzers/test_flower-superexec.sh /ryzers/run-superexec.sh + +RUN mkdir -p /app/certificates + +# When running --plugin-type clientapp this connects out to the SuperNode +# at supernode:9094; when running --plugin-type serverapp it connects out +# to the SuperLink at superlink:9091. No inbound ports are exposed. + +# Default CMD is the role launcher (reads FLOWER_PLUGIN_TYPE etc. from +# config.yaml). To run the install-validation smoke test instead: +# ryzers run /ryzers/test_flower-superexec.sh +CMD /ryzers/run-superexec.sh diff --git a/packages/federated/flower-superexec/README.md b/packages/federated/flower-superexec/README.md new file mode 100644 index 00000000..8f9c8531 --- /dev/null +++ b/packages/federated/flower-superexec/README.md @@ -0,0 +1,116 @@ +# Flower SuperExec Docker Setup + +`flower-superexec` is Flower's process runner. The same binary runs both +the **ServerApp** (server-side aggregation logic) and the **ClientApp** +(client-side training), selected with `--plugin-type {serverapp,clientapp}`. + +This Ryzer is the only one in the federation that actually executes +PyTorch training, so it sits on top of the ROCm/PyTorch base image and +inherits GPU access. It ships with the `quickstart-pytorch` example app +preinstalled at `/app/`. + +A third plugin type, `submit`, runs `flwr run /app local` once and +exits — the `local` federation in `/app/pyproject.toml` points at the +SuperLink ExecApi on `127.0.0.1:9093`. Used by +[`../run-local.sh`](../run-local.sh) to kick off the example run after +everything else is up. The container uses `--network host`, so the +ServerApp/ClientApp/submit roles all reach SuperLink and SuperNode via +loopback in single-machine deployments and via the host network +directly in multi-machine ones. + +## Build + +```sh +ryzers build --name flower-superexec flower-base flower-superexec +``` + +## Run + +SuperExec flags are driven by environment variables declared in +`config.yaml` (with shell-expansion defaults). The role +(`serverapp` vs `clientapp`) is selected via `FLOWER_PLUGIN_TYPE`. + +### Run as ServerApp (server machine) + +```sh +export FLOWER_PLUGIN_TYPE=serverapp +export FLOWER_APPIO_ADDR=127.0.0.1:9091 # local SuperLink +ryzers run +``` + +### Run as ClientApp (client machine) + +```sh +export FLOWER_PLUGIN_TYPE=clientapp +export FLOWER_APPIO_ADDR=127.0.0.1:9094 # local SuperNode +ryzers run +``` + +## Putting it together (mirrors the upstream tutorial) + +**On the server machine** (SuperLink + ServerApp superexec): + +```sh +# --name sets the final image tag; without it both builds would clobber +# the default "ryzerdocker" image and `ryzers run --name ` would +# not find its generated run-script. +ryzers build --name flower-superlink flower-base flower-superlink +ryzers build --name flower-superexec flower-base flower-superexec + +# Terminal 1 — SuperLink +ryzers run --name flower-superlink + +# Terminal 2 — ServerApp superexec +export FLOWER_PLUGIN_TYPE=serverapp +export FLOWER_APPIO_ADDR=127.0.0.1:9091 +ryzers run --name flower-superexec +``` + +**On each client machine** (SuperNode + ClientApp superexec): + +```sh +ryzers build --name flower-supernode flower-base flower-supernode +ryzers build --name flower-superexec flower-base flower-superexec + +# Terminal 1 — SuperNode (connects to remote SuperLink) +export SUPERLINK_IP=192.168.2.33 +export FLOWER_PARTITION_ID=0 +export FLOWER_NUM_PARTITIONS=2 +ryzers run --name flower-supernode + +# Terminal 2 — ClientApp superexec (runs the PyTorch training) +export FLOWER_PLUGIN_TYPE=clientapp +export FLOWER_APPIO_ADDR=127.0.0.1:9094 +ryzers run --name flower-superexec +``` + +Then from any machine with the `flwr` CLI installed: + +```sh +flwr run /app remote-deployment # see config.toml in upstream tutorial +``` + +For TLS, set `FLOWER_INSECURE=0` on each component and stage the certs +generated by `flower-superlink/gen-certs.sh` (see that Ryzer's README). + +### Env-var reference + +| Variable | Default | Purpose | +|----------|---------|---------| +| `FLOWER_PLUGIN_TYPE` | `serverapp` | `serverapp`, `clientapp`, or `submit` | +| `FLOWER_PARTITION_ID` | `0` | For `clientapp`, picks which SuperNode ClientAppIo port to attach to | +| `FLOWER_APPIO_ADDR` | derived | `127.0.0.1:9091` (serverapp) or `127.0.0.1:$((9094 + FLOWER_PARTITION_ID))` (clientapp); ignored for `submit` | +| `FLOWER_INSECURE` | `1` | `1` = `--insecure`; `0` = TLS via `FLOWER_CA_CERT` | + +## Bundled example + +`/app/` contains a copy of [`examples/quickstart-pytorch`](https://github.com/adap/flower/tree/v1.26.1/examples/quickstart-pytorch), +modified to drop the `torch==2.8.0` pin so the ROCm PyTorch from the +base image is used unchanged. + +## References + +- [SuperExec / process isolation](https://flower.ai/docs/framework/how-to-deploy-flower-server-using-process-isolation.html) +- [Multi-machine Docker tutorial](https://flower.ai/docs/framework/docker/tutorial-deploy-on-multiple-machines.html) + +Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. diff --git a/packages/federated/flower-superexec/app/README.md b/packages/federated/flower-superexec/app/README.md new file mode 100644 index 00000000..dc92e10c --- /dev/null +++ b/packages/federated/flower-superexec/app/README.md @@ -0,0 +1,75 @@ +--- +tags: [quickstart, vision, fds] +dataset: [CIFAR-10] +framework: [torch, torchvision] +--- + +# Federated Learning with PyTorch and Flower (Quickstart Example) + +This introductory example to Flower uses PyTorch, but deep knowledge of PyTorch is not necessarily required to run the example. However, it will help you understand how to adapt Flower to your use case. Running this example in itself is quite easy. This example uses [Flower Datasets](https://flower.ai/docs/datasets/) to download, partition and preprocess the CIFAR-10 dataset. + +## Set up the project + +### Fetch the app + +Install Flower: + +```shell +pip install flwr +``` + +Fetch the app: + +```shell +flwr new @flwrlabs/quickstart-pytorch +``` + +This will create a new directory called `quickstart-pytorch` with the following structure: + +```shell +quickstart-pytorch +├── pytorchexample +│ ├── __init__.py +│ ├── client_app.py # Defines your ClientApp +│ ├── server_app.py # Defines your ServerApp +│ └── task.py # Defines your model, training and data loading +├── pyproject.toml # Project metadata like dependencies and configs +└── README.md +``` + +### Install dependencies and project + +Install the dependencies defined in `pyproject.toml` as well as the `pytorchexample` package. + +```bash +pip install -e . +``` + +## Run the project + +You can run your Flower project in both _simulation_ and _deployment_ mode without making changes to the code. If you are starting with Flower, we recommend you using the _simulation_ mode as it requires fewer components to be launched manually. By default, `flwr run` will make use of the Simulation Engine. + +### Run with the Simulation Engine + +> [!TIP] +> This example runs faster when the `ClientApp`s have access to a GPU. If your system has one, you can make use of it by configuring the `backend.client-resources` component in your Flower Configuration. Check the [Simulation Engine documentation](https://flower.ai/docs/framework/how-to-run-simulations.html) to learn more about Flower simulations and how to optimize them. + +```bash +# Run with the default federation (CPU only) +flwr run . +``` + +You can also override some of the settings for your `ClientApp` and `ServerApp` defined in `pyproject.toml`. For example: + +```bash +flwr run . --run-config "num-server-rounds=5 learning-rate=0.05" +``` + +> [!TIP] +> For a more detailed walk-through check our [quickstart PyTorch tutorial](https://flower.ai/docs/framework/tutorial-quickstart-pytorch.html) + +### Run with the Deployment Engine + +Follow this [how-to guide](https://flower.ai/docs/framework/how-to-run-flower-with-deployment-engine.html) to run the same app in this example but with Flower's Deployment Engine. After that, you might be intersted in setting up [secure TLS-enabled communications](https://flower.ai/docs/framework/how-to-enable-tls-connections.html) and [SuperNode authentication](https://flower.ai/docs/framework/how-to-authenticate-supernodes.html) in your federation. + +If you are already familiar with how the Deployment Engine works, you may want to learn how to run it using Docker. Check out the [Flower with Docker](https://flower.ai/docs/framework/docker/index.html) documentation. diff --git a/packages/federated/flower-superexec/app/pyproject.toml b/packages/federated/flower-superexec/app/pyproject.toml new file mode 100644 index 00000000..317a0b46 --- /dev/null +++ b/packages/federated/flower-superexec/app/pyproject.toml @@ -0,0 +1,57 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "quickstart-pytorch" +version = "1.0.1" +description = "Federated Learning with PyTorch and Flower (Quickstart Example)" +license = "Apache-2.0" +# torch and torchvision are intentionally omitted: they are provided by +# the ROCm/PyTorch base image. Pinning them here would cause pip to +# replace the ROCm build with the upstream CUDA/CPU wheel. +# flwr is pinned to exactly the version installed by flower-base +# (FLWR_VERSION in flower-base/Dockerfile). A loose lower bound like +# "flwr>=1.26.0" lets `pip install /app` silently UPGRADE flwr in the +# superexec image to whatever is newest on PyPI at build time, while +# flower-superlink/flower-supernode stay at the pinned version. The +# resulting version skew makes the SuperLink/SuperNode raise cryptic +# server-side errors ("Exception calling application: 'config'" / "'script'") +# on the Fleet handshake. Keep this pin in lockstep with flower-base. +dependencies = [ + "flwr==1.32.0", + "flwr-datasets[vision]>=0.5.0", +] + +[tool.hatch.build.targets.wheel] +packages = ["."] + +[tool.flwr.app] +publisher = "flwrlabs" + +[tool.flwr.app.components] +serverapp = "pytorchexample.server_app:app" +clientapp = "pytorchexample.client_app:app" + +[tool.flwr.app.config] +num-server-rounds = 3 +fraction-evaluate = 0.5 +local-epochs = 1 +learning-rate = 0.1 +batch-size = 32 + +# Flower requires the [tool.flwr.federations] table with a `default` +# key; omitting it makes flwr fall back to its legacy-config migration +# path, which fails with "failed to migrate legacy toml configuration". +[tool.flwr.federations] +default = "local" + +# `local` federation: targets the SuperLink ExecApi on the host loopback +# (works under --network host). Used by `flwr run /app local`, which the +# `submit` plugin type in run-superexec.sh invokes. +[tool.flwr.federations.local] +address = "127.0.0.1:9093" +insecure = true + +# SuperLink default ports: 9091 = ServerAppIo (serverapp superexec), +# 9092 = Fleet (SuperNode), 9093 = ExecApi (flwr run submitter). diff --git a/packages/federated/flower-superexec/app/pytorchexample/__init__.py b/packages/federated/flower-superexec/app/pytorchexample/__init__.py new file mode 100644 index 00000000..d29a98eb --- /dev/null +++ b/packages/federated/flower-superexec/app/pytorchexample/__init__.py @@ -0,0 +1 @@ +"""pytorchexample.""" diff --git a/packages/federated/flower-superexec/app/pytorchexample/client_app.py b/packages/federated/flower-superexec/app/pytorchexample/client_app.py new file mode 100644 index 00000000..95efa9b7 --- /dev/null +++ b/packages/federated/flower-superexec/app/pytorchexample/client_app.py @@ -0,0 +1,82 @@ +"""pytorchexample: A Flower / PyTorch app.""" + +import torch +from flwr.app import ArrayRecord, Context, Message, MetricRecord, RecordDict +from flwr.clientapp import ClientApp + +from pytorchexample.task import Net, load_data +from pytorchexample.task import test as test_fn +from pytorchexample.task import train as train_fn + +# Flower ClientApp +app = ClientApp() + + +@app.train() +def train(msg: Message, context: Context): + """Train the model on local data.""" + + # Load the model and initialize it with the received weights + model = Net() + model.load_state_dict(msg.content["arrays"].to_torch_state_dict()) + device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + model.to(device) + + # Load the data + partition_id = context.node_config["partition-id"] + num_partitions = context.node_config["num-partitions"] + batch_size = context.run_config["batch-size"] + trainloader, _ = load_data(partition_id, num_partitions, batch_size) + + # Call the training function + train_loss = train_fn( + model, + trainloader, + context.run_config["local-epochs"], + msg.content["config"]["lr"], + device, + ) + + # Construct and return reply Message + model_record = ArrayRecord(model.state_dict()) + metrics = { + "train_loss": train_loss, + "num-examples": len(trainloader.dataset), + } + metric_record = MetricRecord(metrics) + content = RecordDict({"arrays": model_record, "metrics": metric_record}) + return Message(content=content, reply_to=msg) + + +@app.evaluate() +def evaluate(msg: Message, context: Context): + """Evaluate the model on local data.""" + + # Load the model and initialize it with the received weights + model = Net() + model.load_state_dict(msg.content["arrays"].to_torch_state_dict()) + device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + model.to(device) + + # Load the data + partition_id = context.node_config["partition-id"] + num_partitions = context.node_config["num-partitions"] + batch_size = context.run_config["batch-size"] + _, valloader = load_data(partition_id, num_partitions, batch_size) + + # Call the evaluation function + eval_loss, eval_acc = test_fn( + model, + valloader, + device, + ) + + # Construct and return reply Message + metrics = { + "eval_loss": eval_loss, + "eval_acc": eval_acc, + "num-examples": len(valloader.dataset), + } + metric_record = MetricRecord(metrics) + content = RecordDict({"metrics": metric_record}) + return Message(content=content, reply_to=msg) diff --git a/packages/federated/flower-superexec/app/pytorchexample/server_app.py b/packages/federated/flower-superexec/app/pytorchexample/server_app.py new file mode 100644 index 00000000..2a6129e5 --- /dev/null +++ b/packages/federated/flower-superexec/app/pytorchexample/server_app.py @@ -0,0 +1,61 @@ +"""pytorchexample: A Flower / PyTorch app.""" + +import torch +from flwr.app import ArrayRecord, ConfigRecord, Context, MetricRecord +from flwr.serverapp import Grid, ServerApp +from flwr.serverapp.strategy import FedAvg + +from pytorchexample.task import Net, load_centralized_dataset, test + +# Create ServerApp +app = ServerApp() + + +@app.main() +def main(grid: Grid, context: Context) -> None: + """Main entry point for the ServerApp.""" + + # Read run config + fraction_evaluate: float = context.run_config["fraction-evaluate"] + num_rounds: int = context.run_config["num-server-rounds"] + lr: float = context.run_config["learning-rate"] + + # Load global model + global_model = Net() + arrays = ArrayRecord(global_model.state_dict()) + + # Initialize FedAvg strategy + strategy = FedAvg(fraction_evaluate=fraction_evaluate) + + # Start strategy, run FedAvg for `num_rounds` + result = strategy.start( + grid=grid, + initial_arrays=arrays, + train_config=ConfigRecord({"lr": lr}), + num_rounds=num_rounds, + evaluate_fn=global_evaluate, + ) + + # Save final model to disk + print("\nSaving final model to disk...") + state_dict = result.arrays.to_torch_state_dict() + torch.save(state_dict, "final_model.pt") + + +def global_evaluate(server_round: int, arrays: ArrayRecord) -> MetricRecord: + """Evaluate model on central data.""" + + # Load the model and initialize it with the received weights + model = Net() + model.load_state_dict(arrays.to_torch_state_dict()) + device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + model.to(device) + + # Load entire test set + test_dataloader = load_centralized_dataset() + + # Evaluate the global model on the test set + test_loss, test_acc = test(model, test_dataloader, device) + + # Return the evaluation metrics + return MetricRecord({"accuracy": test_acc, "loss": test_loss}) diff --git a/packages/federated/flower-superexec/app/pytorchexample/task.py b/packages/federated/flower-superexec/app/pytorchexample/task.py new file mode 100644 index 00000000..f701ceb8 --- /dev/null +++ b/packages/federated/flower-superexec/app/pytorchexample/task.py @@ -0,0 +1,109 @@ +"""pytorchexample: A Flower / PyTorch app.""" + +import torch +import torch.nn as nn +import torch.nn.functional as F +from datasets import load_dataset +from flwr_datasets import FederatedDataset +from flwr_datasets.partitioner import IidPartitioner +from torch.utils.data import DataLoader +from torchvision.transforms import Compose, Normalize, ToTensor + + +class Net(nn.Module): + """Model (simple CNN adapted from 'PyTorch: A 60 Minute Blitz')""" + + def __init__(self): + super(Net, self).__init__() + self.conv1 = nn.Conv2d(3, 6, 5) + self.pool = nn.MaxPool2d(2, 2) + self.conv2 = nn.Conv2d(6, 16, 5) + self.fc1 = nn.Linear(16 * 5 * 5, 120) + self.fc2 = nn.Linear(120, 84) + self.fc3 = nn.Linear(84, 10) + + def forward(self, x): + x = self.pool(F.relu(self.conv1(x))) + x = self.pool(F.relu(self.conv2(x))) + x = x.view(-1, 16 * 5 * 5) + x = F.relu(self.fc1(x)) + x = F.relu(self.fc2(x)) + return self.fc3(x) + + +fds = None # Cache FederatedDataset + +pytorch_transforms = Compose([ToTensor(), Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) + + +def apply_transforms(batch): + """Apply transforms to the partition from FederatedDataset.""" + batch["img"] = [pytorch_transforms(img) for img in batch["img"]] + return batch + + +def load_data(partition_id: int, num_partitions: int, batch_size: int): + """Load partition CIFAR10 data.""" + # Only initialize `FederatedDataset` once + global fds + if fds is None: + partitioner = IidPartitioner(num_partitions=num_partitions) + fds = FederatedDataset( + dataset="uoft-cs/cifar10", + partitioners={"train": partitioner}, + ) + partition = fds.load_partition(partition_id) + # Divide data on each node: 80% train, 20% test + partition_train_test = partition.train_test_split(test_size=0.2, seed=42) + # Construct dataloaders + partition_train_test = partition_train_test.with_transform(apply_transforms) + trainloader = DataLoader( + partition_train_test["train"], batch_size=batch_size, shuffle=True + ) + testloader = DataLoader(partition_train_test["test"], batch_size=batch_size) + return trainloader, testloader + + +def load_centralized_dataset(): + """Load test set and return dataloader.""" + # Load entire test set + test_dataset = load_dataset("uoft-cs/cifar10", split="test") + dataset = test_dataset.with_format("torch").with_transform(apply_transforms) + return DataLoader(dataset, batch_size=128) + + +def train(net, trainloader, epochs, lr, device): + """Train the model on the training set.""" + net.to(device) # move model to GPU if available + criterion = torch.nn.CrossEntropyLoss().to(device) + optimizer = torch.optim.SGD(net.parameters(), lr=lr, momentum=0.9) + net.train() + running_loss = 0.0 + for _ in range(epochs): + for batch in trainloader: + images = batch["img"].to(device) + labels = batch["label"].to(device) + optimizer.zero_grad() + loss = criterion(net(images), labels) + loss.backward() + optimizer.step() + running_loss += loss.item() + avg_trainloss = running_loss / (epochs * len(trainloader)) + return avg_trainloss + + +def test(net, testloader, device): + """Validate the model on the test set.""" + net.to(device) + criterion = torch.nn.CrossEntropyLoss() + correct, loss = 0, 0.0 + with torch.no_grad(): + for batch in testloader: + images = batch["img"].to(device) + labels = batch["label"].to(device) + outputs = net(images) + loss += criterion(outputs, labels).item() + correct += (torch.max(outputs.data, 1)[1] == labels).sum().item() + accuracy = correct / len(testloader.dataset) + loss = loss / len(testloader) + return loss, accuracy diff --git a/packages/federated/flower-superexec/config.yaml b/packages/federated/flower-superexec/config.yaml new file mode 100644 index 00000000..61823583 --- /dev/null +++ b/packages/federated/flower-superexec/config.yaml @@ -0,0 +1,35 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +# gpu_support defaults to true — this is the only flower-* Ryzer that +# actually runs training, so ROCm access matters here. + +# Host networking (and --rm) already come from RYZERS_DEFAULT_RUN_FLAGS, +# so we don't repeat them here — Docker rejects "--network host" twice. +# Host networking lets the superexec reach its paired SuperLink (serverapp +# role) or SuperNode (clientapp role) over 127.0.0.1 on a single host, and +# bind/connect directly on the host network in a distributed deployment. +# +# The label lets run-local.sh reliably clean up flower containers across +# image rebuilds. +docker_extra_run_flags: "--label ryzers-flower-local=1" + +volume_mappings: +- "$PWD/workspace/flower/superlink-certificates:/app/certificates:ro" + +# Consumed by /ryzers/run-superexec.sh. +# +# Plugin types: +# serverapp — runs the ServerApp; talks to SuperLink ServerAppIo (default :9091) +# clientapp — runs a ClientApp; talks to its paired SuperNode (default :9094+PARTITION_ID) +# submit — one-shot `flwr run /app local`, then exits (no daemon) +environment_variables: +- "FLOWER_PLUGIN_TYPE=${FLOWER_PLUGIN_TYPE:-serverapp}" +- "FLOWER_PARTITION_ID=${FLOWER_PARTITION_ID:-0}" +- "FLOWER_INSECURE=${FLOWER_INSECURE:-1}" +# FLOWER_APPIO_ADDR is computed by run-superexec.sh from PLUGIN_TYPE + +# PARTITION_ID by default; export it explicitly to override. +- "FLOWER_APPIO_ADDR=${FLOWER_APPIO_ADDR:-}" +# Strix Point (gfx1150) compatibility shim — uncomment if your iGPU +# isn't directly recognised by the ROCm runtime. +# - "HSA_OVERRIDE_GFX_VERSION=11.0.0" diff --git a/packages/federated/flower-superexec/run-superexec.sh b/packages/federated/flower-superexec/run-superexec.sh new file mode 100644 index 00000000..44ed94b4 --- /dev/null +++ b/packages/federated/flower-superexec/run-superexec.sh @@ -0,0 +1,63 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# Entry point for a flower-superexec process (the runner used for both +# ServerApp and ClientApp roles in Flower's process-isolation model). +# +# Invoke via: +# ryzers run /ryzers/run-superexec.sh +# +# Required env vars: +# FLOWER_PLUGIN_TYPE — "serverapp", "clientapp", or "submit" +# (default serverapp) +# FLOWER_PARTITION_ID — for clientapp, offsets the default +# SuperNode ClientAppIo port (9094 + ID). +# FLOWER_APPIO_ADDR — host:port of the paired SuperLink (ServerApp) +# or SuperNode (ClientApp). Default depends on +# plugin type. +# +# "submit" plugin type is a one-shot helper: it runs `flwr run /app local` +# against the local SuperLink (using the `local` federation pre-baked +# into /app/pyproject.toml) and then exits. + +set -e + +FLOWER_PLUGIN_TYPE="${FLOWER_PLUGIN_TYPE:-serverapp}" +FLOWER_INSECURE="${FLOWER_INSECURE:-1}" +FLOWER_PARTITION_ID="${FLOWER_PARTITION_ID:-0}" + +if [ "${FLOWER_PLUGIN_TYPE}" = "submit" ]; then + # `--stream` keeps this process attached to the run and returns only once + # the run has finished (i.e. the ServerApp has written final_model.pt to + # disk). This lets the local orchestrator (run-local.sh) detect completion + # and tear the federation down afterwards instead of leaving it running. + echo "Submitting: flwr run /app local --stream" + exec flwr run /app local --stream +fi + +case "${FLOWER_PLUGIN_TYPE}" in + serverapp) DEFAULT_ADDR="127.0.0.1:9091" ;; + clientapp) DEFAULT_ADDR="127.0.0.1:$((9094 + FLOWER_PARTITION_ID))" ;; + *) + echo "FLOWER_PLUGIN_TYPE must be 'serverapp', 'clientapp', or 'submit' (got: ${FLOWER_PLUGIN_TYPE})" >&2 + exit 2 + ;; +esac + +FLOWER_APPIO_ADDR="${FLOWER_APPIO_ADDR:-${DEFAULT_ADDR}}" + +ARGS=( + --plugin-type "${FLOWER_PLUGIN_TYPE}" + --appio-api-address "${FLOWER_APPIO_ADDR}" +) + +if [ "${FLOWER_INSECURE}" = "1" ]; then + ARGS+=(--insecure) +else + ARGS+=(--root-certificates "${FLOWER_CA_CERT:-/app/certificates/ca.crt}") +fi + +echo "Starting: flower-superexec ${ARGS[*]}" +exec flower-superexec "${ARGS[@]}" diff --git a/packages/federated/flower-superexec/test.sh b/packages/federated/flower-superexec/test.sh new file mode 100644 index 00000000..79b18585 --- /dev/null +++ b/packages/federated/flower-superexec/test.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# Smoke test: validate that the flower-superexec CLI is available, that +# the bundled quickstart-pytorch app imports cleanly, and that PyTorch +# can see the ROCm GPU. + +set -e + +echo "Running tests for flower-superexec..." + +flower-superexec --help >/dev/null +flower-superexec --help 2>&1 | grep -q -- "--plugin-type" \ + || { echo "FAIL: --plugin-type flag missing from help"; exit 1; } + +python3 - <<'PY' +import torch +import flwr +from pytorchexample import server_app, client_app + +print(f"flwr version: {flwr.__version__}") +print(f"torch version: {torch.__version__} (HIP: {torch.version.hip})") +print(f"torch.cuda.is_available(): {torch.cuda.is_available()}") +assert hasattr(server_app, "app"), "pytorchexample.server_app.app missing" +assert hasattr(client_app, "app"), "pytorchexample.client_app.app missing" +print("quickstart-pytorch ServerApp and ClientApp imported successfully.") +PY + +echo "Tests passed!" diff --git a/packages/federated/flower-superlink/Dockerfile b/packages/federated/flower-superlink/Dockerfile new file mode 100644 index 00000000..5c197f10 --- /dev/null +++ b/packages/federated/flower-superlink/Dockerfile @@ -0,0 +1,30 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +# Install openssl (used by gen-certs.sh) — base image normally has it, but be safe. +RUN apt-get update && apt-get install -y --no-install-recommends \ + openssl \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /ryzers +COPY test.sh /ryzers/test_flower-superlink.sh +COPY gen-certs.sh /ryzers/gen-certs.sh +COPY run-superlink.sh /ryzers/run-superlink.sh +RUN chmod +x /ryzers/test_flower-superlink.sh /ryzers/gen-certs.sh /ryzers/run-superlink.sh + +# Persisted SuperLink state and TLS certificates are expected to be +# mounted from the host (see config.yaml). +RUN mkdir -p /app/state /app/certificates + +# 9091 = ExecApi (control plane: flwr run / serverapp superexec) +# 9092 = FleetApi (data plane: supernodes connect here) +# 9093 = ServerAppIo (legacy / Exec API TLS) +EXPOSE 9091 9092 9093 + +# Default CMD is the role launcher (reads env vars from config.yaml). +# To run the install-validation smoke test instead: +# ryzers run /ryzers/test_flower-superlink.sh +CMD /ryzers/run-superlink.sh diff --git a/packages/federated/flower-superlink/README.md b/packages/federated/flower-superlink/README.md new file mode 100644 index 00000000..93436444 --- /dev/null +++ b/packages/federated/flower-superlink/README.md @@ -0,0 +1,81 @@ +# Flower SuperLink Docker Setup + +The SuperLink is the central coordinator in a Flower deployment. It +accepts connections from the ServerApp superexec (ServerAppIo, port +9091), SuperNodes (Fleet API, port 9092), and the `flwr run` submitter +(Exec API, port 9093). Run state is persisted to `/app/state`. + +This Ryzer runs on the **server machine** and is the first component you +start when bringing up a federation. The container uses `--network host` +so all three ports bind directly on the host — local components reach +them via `127.0.0.1`, remote ones via the server's IP. + +For a single-machine smoke test that brings up SuperLink + ServerApp + +two SuperNode/ClientApp pairs and submits the example run, use +[`../run-local.sh`](../run-local.sh). + +## Build + +```sh +ryzers build --name flower-superlink flower-base flower-superlink +``` + +## Run + +SuperLink flags are driven by environment variables declared in +`config.yaml` (with shell-expansion defaults). Export them in your +shell before `ryzers run` to override. + +### Insecure (local testing) + +```sh +ryzers run +``` + +### With TLS (real deployment) + +1. Generate certs on the server machine, specifying its routable IP: + + ```sh + SUPERLINK_IP=192.168.2.33 \ + bash packages/federated/flower-superlink/gen-certs.sh + ``` + + This produces `ca.crt`, `server.pem`, `server.key` under + `./workspace/flower/superlink-certificates/`, which the Ryzer's + `config.yaml` mounts read-only at `/app/certificates/`. + +2. Copy `ca.crt` to every client machine (the SuperNodes will need it). + +3. Launch the SuperLink with TLS: + + ```sh + export FLOWER_INSECURE=0 + ryzers run + ``` + + To override the cert paths inside the container, also export + `FLOWER_CA_CERT`, `FLOWER_SERVER_CERT`, `FLOWER_SERVER_KEY`. + +### Env-var reference + +| Variable | Default | Purpose | +|----------|---------|---------| +| `FLOWER_INSECURE` | `1` | `1` = `--insecure`; `0` = enable TLS flags | +| `FLOWER_ISOLATION` | `process` | Passed to `--isolation` | +| `FLOWER_STATE_DB` | `/app/state/state.db` | Persisted run state | + +## Ports + +| Port | API | Used by | +|------|-----|---------| +| 9091 | ServerAppIo | local `flower-superexec --plugin-type serverapp` | +| 9092 | Fleet | remote SuperNodes | +| 9093 | Exec | `flwr run` submitter (and the `submit` plugin-type) | + +## References + +- [SuperLink reference](https://flower.ai/docs/framework/ref-api-cli.html#flower-superlink) +- [Multi-machine Docker tutorial](https://flower.ai/docs/framework/docker/tutorial-deploy-on-multiple-machines.html) + +Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. diff --git a/packages/federated/flower-superlink/config.yaml b/packages/federated/flower-superlink/config.yaml new file mode 100644 index 00000000..d929cc57 --- /dev/null +++ b/packages/federated/flower-superlink/config.yaml @@ -0,0 +1,33 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +# SuperLink is the coordinator. It does not run training and does not +# need the GPU or an X display. +gpu_support: false +x11_display: false + +# Host networking (and --rm) already come from RYZERS_DEFAULT_RUN_FLAGS, +# so we don't repeat them here — Docker rejects "--network host" twice. +# Host networking lets local deployments address every component over +# 127.0.0.1 and distributed ones bind ports directly on the host (no -p +# hop). SuperLink listens on host 9091/9092/9093. +# +# The label lets run-local.sh reliably clean up flower containers across +# image rebuilds (an `--filter ancestor=` cleanup misses containers +# whose image was orphaned by a rebuild). +docker_extra_run_flags: "--label ryzers-flower-local=1" + +volume_mappings: +- "$PWD/workspace/flower/state:/app/state" +- "$PWD/workspace/flower/superlink-certificates:/app/certificates:ro" + +# Consumed by /ryzers/run-superlink.sh. Override by exporting the same +# variables in your shell before `ryzers run`. +environment_variables: +- "FLOWER_INSECURE=${FLOWER_INSECURE:-1}" +- "FLOWER_ISOLATION=${FLOWER_ISOLATION:-process}" +- "FLOWER_STATE_DB=${FLOWER_STATE_DB:-/app/state/state.db}" +# Set FLWR_LOG_LEVEL=DEBUG in your shell before `ryzers run` to make the +# SuperLink print full tracebacks (e.g. the server-side KeyError behind a +# gRPC "Exception calling application: ..." seen by SuperNode/superexec). +- "FLWR_LOG_LEVEL=${FLWR_LOG_LEVEL:-INFO}" diff --git a/packages/federated/flower-superlink/gen-certs.sh b/packages/federated/flower-superlink/gen-certs.sh new file mode 100644 index 00000000..fd69d44f --- /dev/null +++ b/packages/federated/flower-superlink/gen-certs.sh @@ -0,0 +1,64 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# Generate a self-signed CA + server certificate for the Flower SuperLink. +# +# Usage: SUPERLINK_IP=192.168.2.33 ./gen-certs.sh [output-dir] +# Default: SUPERLINK_IP=127.0.0.1, output-dir=$PWD/workspace/flower/superlink-certificates +# +# Outputs (matching the upstream tutorial's expected layout): +# /ca.crt — root cert; copy to each client machine +# /server.pem — server cert (SAN includes SUPERLINK_IP) +# /server.key — server private key +# +# After generation, the flower-superlink Ryzer will pick these up via its +# volume mount (config.yaml maps the output dir to /app/certificates:ro). + +set -euo pipefail + +SUPERLINK_IP="${SUPERLINK_IP:-127.0.0.1}" +OUT_DIR="${1:-$PWD/workspace/flower/superlink-certificates}" + +mkdir -p "$OUT_DIR" +cd "$OUT_DIR" + +echo "Generating Flower TLS certs for SUPERLINK_IP=${SUPERLINK_IP} in ${OUT_DIR}" + +# 1. CA +openssl genrsa -out ca.key 4096 +openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \ + -subj "/CN=Flower SuperLink CA" \ + -out ca.crt + +# 2. Server key + CSR +openssl genrsa -out server.key 4096 +openssl req -new -key server.key \ + -subj "/CN=${SUPERLINK_IP}" \ + -out server.csr + +# 3. Server cert signed by the CA, with SAN +cat >server.ext </dev/null + +flower-superlink --insecure & +PID=$! +trap "kill $PID 2>/dev/null || true" EXIT + +# Wait up to 15s for port 9092 to be listening +for i in $(seq 1 15); do + if (echo >/dev/tcp/127.0.0.1/9092) >/dev/null 2>&1; then + echo "SuperLink is listening on 9092" + echo "Tests passed!" + exit 0 + fi + sleep 1 +done + +echo "FAIL: SuperLink did not bind port 9092 within 15s" +exit 1 diff --git a/packages/federated/flower-supernode/Dockerfile b/packages/federated/flower-supernode/Dockerfile new file mode 100644 index 00000000..defbb1b1 --- /dev/null +++ b/packages/federated/flower-supernode/Dockerfile @@ -0,0 +1,20 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +WORKDIR /ryzers +COPY test.sh /ryzers/test_flower-supernode.sh +COPY run-supernode.sh /ryzers/run-supernode.sh +RUN chmod +x /ryzers/test_flower-supernode.sh /ryzers/run-supernode.sh + +RUN mkdir -p /app/certificates + +# ClientAppIo port — a paired superexec (clientapp plugin) connects here. +EXPOSE 9094 + +# Default CMD is the role launcher (reads env vars from config.yaml). +# To run the install-validation smoke test instead: +# ryzers run /ryzers/test_flower-supernode.sh +CMD /ryzers/run-supernode.sh diff --git a/packages/federated/flower-supernode/README.md b/packages/federated/flower-supernode/README.md new file mode 100644 index 00000000..b28cac16 --- /dev/null +++ b/packages/federated/flower-supernode/README.md @@ -0,0 +1,68 @@ +# Flower SuperNode Docker Setup + +The SuperNode runs on each **client machine** in a Flower deployment. It +connects to the SuperLink's Fleet API (port 9092) and exposes a local +ClientAppIo socket (port 9094) for the paired `flower-superexec +--plugin-type clientapp` to attach to. + +Pair this Ryzer with `flower-superexec` on every client host. The +SuperNode does the federation plumbing; the superexec runs the actual +PyTorch/ROCm training. + +The container uses `--network host`, so `SUPERLINK_IP` is the only +network knob you usually need to set. The ClientAppIo socket defaults +to `0.0.0.0:$((9094 + FLOWER_PARTITION_ID))` so multiple SuperNodes can +share a host for local testing without colliding (see +[`../run-local.sh`](../run-local.sh)). + +## Build + +```sh +ryzers build --name flower-supernode flower-base flower-supernode +``` + +## Run + +SuperNode flags are driven by environment variables declared in +`config.yaml` (with shell-expansion defaults). Export them in your +shell before `ryzers run` to override. + +### Insecure (testing) + +```sh +export SUPERLINK_IP=192.168.2.33 +export FLOWER_PARTITION_ID=0 +export FLOWER_NUM_PARTITIONS=2 +ryzers run +``` + +### With TLS + +Place `ca.crt` at `./workspace/flower/superlink-certificates/ca.crt` +(the volume mount in `config.yaml` exposes it at `/app/certificates/ca.crt`), +then: + +```sh +export SUPERLINK_IP=192.168.2.33 +export FLOWER_INSECURE=0 +export FLOWER_PARTITION_ID=0 +export FLOWER_NUM_PARTITIONS=2 +ryzers run +``` + +### Env-var reference + +| Variable | Default | Purpose | +|----------|---------|---------| +| `SUPERLINK_IP` | `127.0.0.1` | Routable IP of the SuperLink host | +| `FLOWER_INSECURE` | `1` | `1` = `--insecure`; `0` = TLS via `FLOWER_CA_CERT` | +| `FLOWER_PARTITION_ID` | `0` | Unique partition for this node (0..N-1) | +| `FLOWER_NUM_PARTITIONS` | `2` | Total clients across the federation | +| `FLOWER_CLIENTAPPIO` | `0.0.0.0:$((9094 + FLOWER_PARTITION_ID))` | Local socket the paired ClientApp connects to (auto-offset for co-located nodes) | +| `FLOWER_ISOLATION` | `process` | Passed to `--isolation` | + +## References + +- [SuperNode reference](https://flower.ai/docs/framework/ref-api-cli.html#flower-supernode) + +Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. diff --git a/packages/federated/flower-supernode/config.yaml b/packages/federated/flower-supernode/config.yaml new file mode 100644 index 00000000..a99eb956 --- /dev/null +++ b/packages/federated/flower-supernode/config.yaml @@ -0,0 +1,33 @@ +# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT + +# The SuperNode itself does not run training; it just orchestrates the +# paired ClientApp superexec. GPU support is left enabled (default) so +# colocating with the superexec is straightforward. + +# Host networking (and --rm) already come from RYZERS_DEFAULT_RUN_FLAGS, +# so we don't repeat them here — Docker rejects "--network host" twice. +# Host networking lets us run multiple instances on one box (each on a +# different ClientAppIo port) for local testing, and lets distributed +# deployments skip explicit port mapping. +# +# The label lets run-local.sh reliably clean up flower containers across +# image rebuilds. +docker_extra_run_flags: "--label ryzers-flower-local=1" + +volume_mappings: +- "$PWD/workspace/flower/superlink-certificates:/app/certificates:ro" + +# Consumed by /ryzers/run-supernode.sh. Override by exporting the same +# variables in your shell before `ryzers run`. +# +# Local (single-machine) testing: defaults are correct — SUPERLINK_IP +# stays 127.0.0.1, FLOWER_CLIENTAPPIO auto-offsets by PARTITION_ID so +# multiple SuperNodes don't collide on port 9094. +# Distributed testing: export SUPERLINK_IP=. +environment_variables: +- "SUPERLINK_IP=${SUPERLINK_IP:-127.0.0.1}" +- "FLOWER_INSECURE=${FLOWER_INSECURE:-1}" +- "FLOWER_PARTITION_ID=${FLOWER_PARTITION_ID:-0}" +- "FLOWER_NUM_PARTITIONS=${FLOWER_NUM_PARTITIONS:-2}" +- "FLOWER_CLIENTAPPIO=${FLOWER_CLIENTAPPIO:-0.0.0.0:$((9094 + ${FLOWER_PARTITION_ID:-0}))}" diff --git a/packages/federated/flower-supernode/run-supernode.sh b/packages/federated/flower-supernode/run-supernode.sh new file mode 100644 index 00000000..ba082607 --- /dev/null +++ b/packages/federated/flower-supernode/run-supernode.sh @@ -0,0 +1,37 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# Entry point for a SuperNode. Invoke via: +# ryzers run /ryzers/run-supernode.sh +# +# Required: +# SUPERLINK_IP — IP of the remote SuperLink (defaults to 127.0.0.1) +# FLOWER_PARTITION_ID — this node's partition (default 0) +# FLOWER_NUM_PARTITIONS — total nodes in the federation (default 2) + +set -e + +SUPERLINK_IP="${SUPERLINK_IP:-127.0.0.1}" +FLOWER_INSECURE="${FLOWER_INSECURE:-1}" +FLOWER_PARTITION_ID="${FLOWER_PARTITION_ID:-0}" +FLOWER_NUM_PARTITIONS="${FLOWER_NUM_PARTITIONS:-2}" +FLOWER_ISOLATION="${FLOWER_ISOLATION:-process}" +FLOWER_CLIENTAPPIO="${FLOWER_CLIENTAPPIO:-0.0.0.0:9094}" + +ARGS=( + --superlink "${SUPERLINK_IP}:9092" + --clientappio-api-address "${FLOWER_CLIENTAPPIO}" + --isolation "${FLOWER_ISOLATION}" + --node-config "partition-id=${FLOWER_PARTITION_ID} num-partitions=${FLOWER_NUM_PARTITIONS}" +) + +if [ "${FLOWER_INSECURE}" = "1" ]; then + ARGS+=(--insecure) +else + ARGS+=(--root-certificates "${FLOWER_CA_CERT:-/app/certificates/ca.crt}") +fi + +echo "Starting: flower-supernode ${ARGS[*]}" +exec flower-supernode "${ARGS[@]}" diff --git a/packages/federated/flower-supernode/test.sh b/packages/federated/flower-supernode/test.sh new file mode 100644 index 00000000..d7e06417 --- /dev/null +++ b/packages/federated/flower-supernode/test.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# Smoke test: verify the flower-supernode binary is installed and the +# --help output renders. We do NOT attempt to connect to a SuperLink in +# the test, since that would require a live federation. + +set -e + +echo "Running tests for flower-supernode..." + +flower-supernode --help >/dev/null + +# Sanity-check that the expected flags are documented +flower-supernode --help 2>&1 | grep -q -- "--superlink" \ + || { echo "FAIL: --superlink flag missing from help"; exit 1; } +flower-supernode --help 2>&1 | grep -q -- "--clientappio-api-address" \ + || { echo "FAIL: --clientappio-api-address flag missing from help"; exit 1; } + +echo "flower-supernode CLI looks good." +echo "Tests passed!" diff --git a/packages/federated/run-local.sh b/packages/federated/run-local.sh new file mode 100755 index 00000000..afd8a07c --- /dev/null +++ b/packages/federated/run-local.sh @@ -0,0 +1,286 @@ +#!/bin/bash +# Copyright (C) 2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +# +# One-shot single-machine smoke test for the Flower federated-learning +# Ryzers. Each long-running component (SuperLink, ServerApp superexec, +# and one SuperNode + ClientApp superexec per partition) is launched in +# its own terminal window so you can watch them individually. Once the +# federation is up, the quickstart-pytorch run is submitted from this +# terminal. +# +# Multi-machine deployment uses the same three Ryzers — point +# SUPERLINK_IP at the server and run the appropriate role on each box. +# See each Ryzer's README for the distributed flow. +# +# Usage: +# cd packages/federated +# ./run-local.sh # 2 partitions, auto-pick terminal +# FLOWER_NUM_PARTITIONS=4 ./run-local.sh # 4 partitions +# RYZERS_TERMINAL=gnome-terminal ./run-local.sh # force a terminal emulator + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +NUM_PARTITIONS="${FLOWER_NUM_PARTITIONS:-2}" + +cd "${REPO_ROOT}" + +# --------------------------------------------------------------------------- +# Pick a terminal emulator. xterm is preferred because it honours $DISPLAY +# reliably over SSH X-forwarding; the others are tried as fallbacks. +# --------------------------------------------------------------------------- +if [ -z "${DISPLAY}" ]; then + echo "ERROR: \$DISPLAY is not set — can't open terminal windows." >&2 + echo " Run over X-forwarding (ssh -X) or set DISPLAY." >&2 + exit 1 +fi + +TERM_EMU="${RYZERS_TERMINAL:-}" +if [ -z "${TERM_EMU}" ]; then + for t in xterm gnome-terminal xfce4-terminal konsole x-terminal-emulator; do + if command -v "${t}" >/dev/null 2>&1; then + TERM_EMU="${t}" + break + fi + done +fi +if [ -z "${TERM_EMU}" ] || ! command -v "${TERM_EMU}" >/dev/null 2>&1; then + echo "ERROR: no terminal emulator found (tried xterm, gnome-terminal, ...)." >&2 + echo " Install one or set RYZERS_TERMINAL." >&2 + exit 1 +fi +echo "Using terminal emulator: ${TERM_EMU}" + +LOG_DIR="$(mktemp -d -t flower-local-XXXXXX)" +echo "Per-component launch scripts: ${LOG_DIR}" + +# spawn TITLE "shell-command" +# Writes a small wrapper script (so we avoid cross-emulator quoting +# hell), then opens it in a new terminal window that stays open after +# the component exits. +spawn() { + local title="$1" + local body="$2" + local script="${LOG_DIR}/${title}.sh" + cat > "${script}" < /dev/tcp/127.0.0.1/"${port}") 2>/dev/null; then + return 0 + fi + if ((i % 10 == 0)); then + echo " ... still waiting for 127.0.0.1:${port} (${i}/${timeout}s)" + fi + sleep 1 + done + return 1 +} + +port_in_use() { + # 0 (true) if something is already listening on 127.0.0.1:. + local port="$1" + (echo > /dev/tcp/127.0.0.1/"${port}") 2>/dev/null +} + +# require_ports_free PORT... +# Abort if any of the given ports already has a listener. Because gRPC +# uses SO_REUSEPORT, a leftover listener would be silently co-bound to by +# the component we are about to start — the exact failure mode this script +# guards against — so we fail fast with diagnostics instead of starting on +# top of it. `wait_for_port` cannot catch this: it treats a stale listener +# as "the service is up". +require_ports_free() { + local p busy=() + for p in "$@"; do + if port_in_use "${p}"; then + busy+=("${p}") + fi + done + if ((${#busy[@]} > 0)); then + echo "ERROR: these ports are still in use after cleanup: ${busy[*]}" >&2 + echo " A stale listener here would be silently co-bound via" >&2 + echo " SO_REUSEPORT and answer some calls with errors like" >&2 + echo " \"Exception calling application: 'config'\"." >&2 + echo >&2 + echo " Find and remove what is holding them, e.g.:" >&2 + echo " docker ps -a --format '{{.ID}} {{.Image}} {{.Names}}'" >&2 + for p in "${busy[@]}"; do + echo " ss -tlnp 'sport = :${p}' # (or: lsof -iTCP:${p} -sTCP:LISTEN)" >&2 + done + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Clean up stale containers FIRST, before building. A rebuild moves the +# flower-* tags onto new image IDs and orphans the old ones, so an +# `--filter ancestor=` cleanup run *after* the build would no longer +# match containers from the previous run (they still reference the old +# image ID) and they'd keep holding the 909x ports. +# +# This MUST be thorough. Every component runs with `--network host`, and +# gRPC enables SO_REUSEPORT by default (flwr does not disable it), so a +# stale SuperLink/SuperNode from a previous run can silently CO-BIND 909x +# alongside the freshly started one. The kernel then load-balances +# connections across both, and the older / half-broken instance answers +# some calls with cryptic gRPC errors such as +# "Exception calling application: 'config'" (a server-side KeyError). +# That is invisible in the new SuperLink's window (it starts fine), which +# makes it very hard to diagnose — so we remove flower containers by EVERY +# signal we have, not just the label. +# --------------------------------------------------------------------------- +echo "== Cleaning up stale containers ==" +# 1) Primary: the label set via each role's docker_extra_run_flags +# (survives image rebuilds). +docker ps -aq --filter "label=ryzers-flower-local=1" | xargs -r docker rm -f >/dev/null 2>&1 || true +# 2) Fallback: anything whose image references a flower-* tag (or the +# default "ryzerdocker" tag) — for containers created before the label +# existed, or built from a differently-tagged cached base. +for name in flower-superlink flower-supernode flower-superexec flower-base ryzerdocker; do + docker ps -aq --filter "ancestor=${name}" | xargs -r docker rm -f >/dev/null 2>&1 || true +done +# 3) Last resort: any remaining container whose image name contains +# "flower" (catches odd tags from earlier iterations). `--filter` has no +# image wildcard, so match on the formatted list instead. +docker ps -a --format '{{.ID}} {{.Image}}' \ + | awk 'tolower($2) ~ /flower/ {print $1}' \ + | xargs -r docker rm -f >/dev/null 2>&1 || true + +# --------------------------------------------------------------------------- +# Remove stale generated run-scripts BEFORE building. `ryzers run` does NOT +# regenerate `ryzers.run..sh` — it blindly `bash`-executes whatever is +# already in the cwd (runner.py). `ryzers build` overwrites the three current +# scripts, but a script left over from an earlier package layout or image +# name is never touched and can be picked up by a stray `ryzers run`, +# reintroducing the cryptic gRPC "Exception calling application: 'config'" / +# "'script'" failures. Wipe them so every run starts from freshly generated +# scripts. +# --------------------------------------------------------------------------- +echo "== Removing stale ryzers run-scripts ==" +rm -f "${REPO_ROOT}"/ryzers.run.*.sh 2>/dev/null || true + +# --------------------------------------------------------------------------- +# Build the three role images. NB: `ryzers build` names the *final* image +# after --name (default "ryzerdocker"), NOT after the last package — so +# --name is required here, otherwise all three would clobber the same +# "ryzerdocker" image and `ryzers run --name ` would not find its +# generated run-script. +# --------------------------------------------------------------------------- +echo "== Building Ryzers ==" +ryzers build --name flower-superlink flower-base flower-superlink +ryzers build --name flower-supernode flower-base flower-supernode +ryzers build --name flower-superexec flower-base flower-superexec + +# Start from a clean SuperLink state. A partial run left over from a +# previous (e.g. failed) submit can make the SuperLink raise +# KeyError('config') -> "Exception calling application: 'config'" when a +# SuperNode connects. This matches the volume mount in +# flower-superlink/config.yaml ($PWD/workspace/flower/state). +echo "== Resetting SuperLink state ==" +rm -rf "${REPO_ROOT}/workspace/flower/state"/* 2>/dev/null || true + +# Confirm cleanup actually freed every host port we are about to bind. +# 9091 ServerAppIo, 9092 Fleet, 9093 ExecApi, plus one ClientAppIo per +# partition (9094 + i). If any is still held, abort before we start — a +# survivor would be co-bound via SO_REUSEPORT and intermittently serve +# stale responses. +echo "== Verifying ports are free ==" +PORTS_TO_CHECK=(9091 9092 9093) +for ((i = 0; i < NUM_PARTITIONS; i++)); do + PORTS_TO_CHECK+=($((9094 + i))) +done +require_ports_free "${PORTS_TO_CHECK[@]}" +echo " All required ports are free: ${PORTS_TO_CHECK[*]}" + +echo "== Starting SuperLink ==" +spawn "flower-superlink" "ryzers run --name flower-superlink" +if ! wait_for_port 9092 120; then + echo "ERROR: SuperLink Fleet API (9092) never came up." >&2 + echo " Check the 'flower-superlink' terminal window for the error." >&2 + exit 1 +fi +echo " SuperLink up (9092 bound)." + +echo "== Starting ServerApp superexec ==" +spawn "flower-serverapp" \ + "FLOWER_PLUGIN_TYPE=serverapp FLOWER_APPIO_ADDR=127.0.0.1:9091 ryzers run --name flower-superexec" + +for ((i = 0; i < NUM_PARTITIONS; i++)); do + echo "== Starting SuperNode partition ${i} ==" + spawn "flower-supernode-${i}" \ + "FLOWER_PARTITION_ID=${i} FLOWER_NUM_PARTITIONS=${NUM_PARTITIONS} ryzers run --name flower-supernode" + + echo "== Starting ClientApp superexec partition ${i} ==" + spawn "flower-clientapp-${i}" \ + "FLOWER_PLUGIN_TYPE=clientapp FLOWER_PARTITION_ID=${i} FLOWER_APPIO_ADDR=127.0.0.1:$((9094 + i)) ryzers run --name flower-superexec" +done + +echo "== Waiting for SuperLink ExecApi (9093) ==" +if ! wait_for_port 9093 120; then + echo "ERROR: SuperLink ExecApi (9093) never came up." >&2 + exit 1 +fi + +# Give the SuperNodes a beat to finish the Fleet handshake before +# submitting — otherwise `flwr run` can race node registration. +echo " Waiting for SuperNodes to register..." +sleep 8 + +# The submit uses `flwr run ... --stream`, so this call blocks until the run +# finishes and the ServerApp has written final_model.pt to disk. +echo "== Submitting quickstart-pytorch run (streaming until complete) ==" +FLOWER_PLUGIN_TYPE=submit ryzers run --name flower-superexec + +# Run finished — tear the whole federation down so every per-component window +# closes on its own. Dropping the sentinel tells each spawned wrapper to exit +# (closing its terminal) instead of waiting on a keypress; killing the +# containers makes each wrapper's `docker run` return so it reaches that check. +echo +echo "== Run complete. Shutting down all components... ==" +touch "${LOG_DIR}/.shutdown" +docker ps -q --filter "label=ryzers-flower-local=1" | xargs -r docker kill >/dev/null 2>&1 || true + +echo "== Done. Model written to disk; all component windows are closing. ==" diff --git a/ryzers/__init__.py b/ryzers/__init__.py index 7335304b..9be24ac1 100644 --- a/ryzers/__init__.py +++ b/ryzers/__init__.py @@ -3,7 +3,7 @@ import os -RYZERS_DEFAULT_INIT_IMAGE = "rocm/pytorch:rocm7.2.2_ubuntu24.04_py3.12_pytorch_release_2.10.0" +RYZERS_DEFAULT_INIT_IMAGE = "rocm/pytorch:rocm7.14_ubuntu26.04_py3.14_pytorch_release_2.12.0" RYZERS_DEFAULT_RUN_FLAGS = "-it --rm --shm-size 16G --cap-add=SYS_PTRACE --network=host --ipc=host" # Auto-detect packages path in editable mode