Skip to content

docs(rocm): publish G10/G11 200-step RL results - #400

Merged
Flink-ddd merged 3 commits into
testfrom
docs/rocm-g10-g11-200-results
Sep 10, 2026
Merged

Flink-ddd merged 3 commits into
testfrom
docs/rocm-g10-g11-200-results

Conversation

@inaniloquentee

@inaniloquentee inaniloquentee commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

What this PR adds

  • Publishes the completed ROCm Qwen3-8B 200-step G10 production P/P and G11 RL-Kernel R/R consistency experiment derived from PR perf(rocm): MFMA batch-invariant GEMM and chunked Triton attention for strict R/R #396.
  • Adds the complete paired-step CSV, structured summary, strict-validation summary, and two reproducible consistency figures.
  • Adds local-log regeneration and W&B upload scripts so the complete RL history and source artifacts can be reproduced and published without rerunning training.

Completed experiment scope

Group Attention / FFN / logp route Ray submission Result
G10 production / production (P/P) raysubmit_Rj3ZWxq3mphM2bFL SUCCEEDED, 200/200
G11 RL-Kernel / RL-Kernel (R/R), strict consistency raysubmit_twpgQ7pxvniaXWXC SUCCEEDED, 200/200

Both runs used the same frozen workload and source configuration. Only the intended G10 P/P versus G11 R/R implementation routes and identity-derived paths differ.

Item Matched value
Hardware 1 node, 8x AMD Instinct MI300X 192GB
Actor parallelism TP4 / CP2 / PP1
Rollout engines 2 engines, TP4 each
Horizon 200 rollout/training steps
Seeds training 1234, rollout 1234
Sampling 1 prompt x 8 samples = global batch 8
Response limit 7,168 tokens
Dynamic batching maximum 4,096 tokens/GPU
vLLM memory utilization 0.38
Rollout log probabilities enabled
Reference model / KL loss enabled, coefficient 0.001
Torch profiler disabled

Version provenance

Arm RL-Kernel VIME Megatron-LM Notes
G10 7a9f3b5 c80200e 1dcf0da Native P/P route
G11 7a9f3b5 c80200e 1dcf0da Strict RL-Kernel R/R route
Result bundle 86877e4 - - CSV, JSON, Markdown, two consistency plots, and upload tooling

Strict train/rollout consistency and reward

  • G11 passes strict validation across all 200 steps: mismatch_count == 0, max_abs_diff == 0, and torch.equal == true over 9,400,614 compared elements from 1,600 samples.
  • G10 completed all 200 steps. Its final validator only rejects missing RL-Kernel operator readbacks, which the native P/P route does not emit; the training result itself is complete.
  • Frozen-input and frozen-source audits match across both arms.

Training and bitwise consistency

Mean train/rollout logp difference

Mean absolute train/rollout logp difference

Data integrity and method

  • Both Ray submissions completed successfully with 200/200 recorded rollout and training steps.
  • The paired CSV contains every available scalar RL, training, consistency, and timing field from both launcher logs. Required fields are finite; no imputation or row deletion is used.
  • The structured summary preserves the full 200-step arithmetic view, a rollout-0-warmup-excluded token-normalized view, and the equal-length matched-step cross-check.
  • The raw launcher logs and full validation JSON are included by the W&B uploader as source artifacts.

Reproducible artifacts

Validation

  • Pre-commit formatting, import, and lint checks pass for both Python scripts.
  • A clean regeneration from the two authoritative launcher logs produces only the two consistency PNG files by default.
  • Performance plots are generated only with --include-performance-plots for external W&B reporting.
  • The W&B uploader completed an offline end-to-end dry run for G10, G11, and the combined report artifact.

@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 45c4ac47-d351-48fb-a224-28db1c083821

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: lamentropetion <3051000145@qq.com>
@inaniloquentee
inaniloquentee force-pushed the docs/rocm-g10-g11-200-results branch from d2eb726 to fba5a35 Compare September 10, 2026 08:21
Signed-off-by: lamentropetion <3051000145@qq.com>

@Flink-ddd Flink-ddd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxiaosong1124
maxiaosong1124 self-requested a review September 10, 2026 09:16

@maxiaosong1124 maxiaosong1124 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Signed-off-by: lamentropetion <3051000145@qq.com>
@Flink-ddd
Flink-ddd merged commit f9d7504 into test Sep 10, 2026
6 of 8 checks passed
@Flink-ddd Flink-ddd added the platform: rocm Specific tasks specific to AMD graphics cards (such as CK, bpreshuffle/FA) label Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

platform: rocm Specific tasks specific to AMD graphics cards (such as CK, bpreshuffle/FA)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants