Multi-agent reinforcement learning for high-dimensional, parametric, and distributed systems

HypeMARL is an efficient decentralized MARL algorithm tailored to the control of high-dimensional, parametric, and distributed systems. Specifically, HypeMARL leverages hypernetworks to effectively parametrize the agents’ policies and value functions with respect to

• the system parameters, improving generalizaion capabilities across multiple scenarios,
• the sinusoidal positional encoding of the agents' relative positions, enhancing spatial awarness and coordination among agents, which is essential to achieve collective non-local behavior of the agents.

Moreover, we propose a model-based extension of the algorithm (MB-HypeMARL) relying on efficient deep learning-based surrogate models approximating the local dynamics to enhance sample efficiency, with a significant reduction of real environments interactions and a minimal deterioration of the policy performance.

HypeMARL and MB-HypeMARL effectively control high-dimensional systems through a collective behavior of the agents, synthesizing parametric and position-aware policies with minimal hyperparameter tuning, and outperforming state-of-the-art decentralized MARL algorithms.

Quickstart

The file run.py allows to run and reproduce all the experiments in the paper. One can simply choose the --agent-type and the --env-name and the code does the rest.

python run.py --agent-type={RL, MARL, HypeMARL, MB-HypeMARL} --env-name={DensityControlVacuum, DensityControlFluid, FlowControlJets, FlowControlMorphingFront}

Getting started

The required packages are listed in the environment.yml file and may be installed in few minutes through the command

conda env create -f environment.yml

The test cases required FEniCS to generate and handle function data. Click here for installation instructions.

Code structure

𝐇𝐲𝐩𝐞𝐌𝐀𝐑𝐋
│
│   README.md
│   requirements.txt    
│   run.py  # Training loops for the different test cases
│   
└───agents
│   │   HypeMARL.py
│   │   MARL.py
│   └───nn
│       │   hyper_network.py
│       │   hyper_policy.py
│       │   hyper_q_network.py
│   
└───env
    │   densitycontrol.py  # Environments for the density control test cases
    │   flowcontrol.py  # Environments for the flow control test cases

Density control in a vacuum

DensityControl_Vacuum in envs/densitycontrol.py implements the environment for the density control in a vacuum test case where we move a Gaussian density function from a random initial mean position to a parametric target destination by controlling the source term of the Fokker-Planck equation.

Density control in a fluid

DensityControl_Fluid in envs/densitycontrol.py implements the environment for the density control in a fluid test case where we move a Gaussian density function from a random initial mean position to a parametric target destination by controlling the source term of the Fokker-Planck equation with an external parametric transport term.

Flow control with vertical jets

FlowControl_Jets in envs/flowcontrol.py implements the environment for the fluid control with vertical jets test case where we maximize (minimize) the pressure at the bottom (top) of the obstacle by controlling the intensity of vertical jets on the obstacle boundary, which is a boundary control for the Navier-Stokes equations.

Flow control with shape morphing

FlowControl_MorphingFront in envs/flowcontrol.py implements the environment for the fluid control with shape morphin test case where we maximize (minimize) the pressure at the bottom (top) of the obstacle by controlling the intensity of vertical jets on the obstacle boundary, which is a boundary control for the Navier-Stokes equations.

Cite

If you use this code for your work, please cite

@misc{hypemarl,
      title={HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems}, 
      author={Nicolò Botteghi and Matteo Tomasetto and Urban Fasel and Francesco Braghin and Andrea Manzoni},
      year={2025},
      eprint={2509.16709},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2509.16709}, 
}