[andrej]

Motivation

IRON provides an alternative for open-source developers to implement end-to-end models. FastFlowLM's flm binary provides a user-friendly interface to interact with models, such as pulling and serving models. By adding an IRON-backend to flm, users can use the same flm commands to run IRON's model implementations, similarly to the RyzenAI Software backend. As a first step, the IRON backend is offered as an alternative for the Llama3.2 1B model and as the main implementation for a new EmbeddingGemma-2 model that was not previously supported by FastFlowLM.

Technical Details

The user/developer is expected to install IRON according to the installation instruction themselves.
When FLM is built with the FLM_ENABLE_IRON CMake flag, it contains the new IRON backend. IRON runs as a Python server subprocess, and FLM communicates with it through a pipe.
There is a very simple JSON-based protocol between the two.
The FLM part of the implementation contains no Python. IRON itself is written in Python so runs as a Python interpreter process; all of the tensor management, dispatch, model graph logic is Python, so cannot be easily ported.

Test Plan

A simple infrastructure test testing that the IRON backend correctly communicates through the JSON protocol is included. Correctness on the IRON side is tested in the IRON project; this only tests that FLM sends the correct kinds of requests to IRON.
The IRON repository counterpart of this will contain an end-to-end test that also builds FLM and makes sure the integration works.

Test Result

Pass.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.