Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1183,6 +1183,39 @@ KEEP OR REJECT

MIInfer succeeds only if its performance claims survive that loop.

---

# 35. Parallel MI50 development

During the current single-MI50 roadmap, two independent MI50 hosts may be used
to run separate goals in parallel:

- `mi50-a`: HP Z840 + MI50 32 GB
- `mi50-b`: Machinist X99 + MI50 32 GB

They are independent experiment hosts, not a distributed inference runtime.

Before parallel work begins, both hosts must use the same qualified MIInfer
userspace environment: exact source/base SHA, OCI image digest, ROCm/HIP
userspace and compiler, build configuration, model artifact/hash, benchmark
inputs, and relevant environment variables. Host-specific CPU, motherboard,
PCIe, kernel, and cooling differences must still be recorded.

Parallel-agent rules:

1. Each agent receives one goal, one host, one base SHA, and one branch.
2. Performance baseline and candidate measurements must run on the same host.
3. Do not compare a candidate on one host against a baseline on the other.
4. Agents must not independently change a shared architectural contract. Freeze
the contract first, then branch parallel work from that commit.
5. Stop when the assigned goal's evidence/exit gate is satisfied; do not
opportunistically start adjacent roadmap work.
6. Reproduce only important milestone-closing or surprising results on the
second host; do not duplicate every exploratory run.

See [`docs/parallel-development.md`](docs/parallel-development.md) for the
current two-host workflow.

## graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.
Expand Down
115 changes: 115 additions & 0 deletions docs/parallel-development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Parallel MI50 Development

## Purpose

MIInfer currently has two independent single-MI50 development hosts:

- **mi50-a:** HP Z840 + AMD Instinct MI50 32 GB
- **mi50-b:** Machinist X99 + AMD Instinct MI50 32 GB

The purpose of this setup is to finish the single-MI50 roadmap faster before
both GPUs are moved into one host for V3. The two machines are not a distributed
inference target.

## First prerequisite — reproducible environment

Do not start parallel M29 implementation until both hosts can run the same
qualified MIInfer environment.

Use one OCI image that works with Podman or Docker. Prefer pinning by image
digest rather than relying only on a mutable tag. The image should contain the
userspace pieces that must remain identical between hosts, including:

- ROCm/HIP userspace and compiler/toolchain required by MIInfer;
- build dependencies and build configuration;
- MIInfer runtime dependencies;
- benchmark/test tooling;
- the expected environment variables and runtime configuration.

The model remains an external artifact but must have the same SHA-256 on both
hosts. Benchmark inputs must also be identical.

The container does **not** make the physical machines identical. Record
host-specific kernel, CPU, motherboard/PCIe, cooling, clocks, power and
temperature information with benchmark evidence.

### Environment-parity exit gate

Both hosts must independently:

1. identify one MI50 32 GB / gfx906 device;
2. build the same MIInfer source SHA from the same OCI image digest;
3. verify the exact model SHA-256;
4. pass the same host/GPU correctness smoke;
5. run a small known benchmark matrix without errors;
6. capture the environment and hardware state used for the run.

Absolute timings do not have to match. The purpose is to prove that both hosts
can execute the same experiment contract reproducibly.

## Parallel work rules

After the environment gate passes, both hosts are peers. Assign whichever
dependency-ready goal provides the most value.

Every Codex workstream receives a small goal packet:

```text
GOAL
HOST
BASE SHA
BRANCH
QUESTION
AUTHORIZED SCOPE
DO NOT TOUCH
REQUIRED EVIDENCE
EXIT CONDITION
```

Performance comparisons are always same-host A/B:

```text
mi50-a: baseline A -> candidate A -> delta A
mi50-b: baseline B -> candidate B -> delta B
```

Never use a baseline from one host to judge a candidate measured on the other.

Parallel implementation is allowed only when dependencies are clear. If two
tasks share an architectural interface, first land/freeze that contract, then
branch both tasks from the contract commit.

Agents stop at their assigned exit condition. They do not continue into the
next roadmap item without a new goal.

## Initial parallel split

Once environment parity passes:

```text
mi50-a
M29.0 — current MIInfer long-context baseline

mi50-b
M29.0A — current gfx906 external-reference calibration
```

When M29.0A closes, mi50-b can move to M29.1 HIP-VMM feasibility while mi50-a
continues any remaining long-context baseline work.

Later work should be parallelized only where the current roadmap dependencies
allow it. Important milestone-closing or surprising results should be reproduced
on the second host before final acceptance; routine exploratory work does not
need duplicate execution.

## End of the two-host phase

Keep both MI50s in separate machines through the single-MI50 roadmap:

```text
M29 -> M30 -> M31
```

After M31 freezes the qualified N=1 baseline and the critical gates have been
reproduced across the two hosts, move both MI50s into one machine and begin V3
dual-MI50 topology qualification.
8 changes: 8 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ entire sequence. Every stage begins through a focused, evidence-backed goal.
The immediate first engineering goal after M29-0000 is M29-0001, covering the
M29.0 long-context baseline and M29.1 physical-backing feasibility.

Before M29 work is split across the two available MI50 hosts, establish one
reproducible OCI-based gfx906 development environment on both the HP Z840 and
Machinist X99 systems. The container must pin the MIInfer userspace/toolchain
configuration and model/benchmark inputs while host-specific hardware and kernel
properties remain recorded evidence. Once both hosts pass the same smoke and
baseline checks, dependency-independent M29 goals may run in parallel. See
[`parallel-development.md`](parallel-development.md).

## M29 — Persistent Context Architecture

**Mission:** Build a placement-independent persistent context substrate,
Expand Down
Loading