ci(benchmarks): add Devstral 2 to H200 nightly benchmarks

[smfirmin]

Description

Summary

Add mistralai/Devstral-2-123B-Instruct-2512 to the nightly benchmark coverage on H200 for both sglang and vllm.

Changes

• add Devstral 2 to MODEL_SPECS with nightly benchmark metadata
• set Devstral 2 tensor parallelism to 4
• register nightly benchmark test classes for Devstral 2
• add Devstral 2 to the single-worker-h200 workflow matrix
• place the workflow entry immediately after the commented-out minimax-m2 benchmark entry

Notes

• this enables nightly H200 runs for both sglang and vllm

Summary by CodeRabbit

Release Notes

• New Features

  • Added support for mistralai/Devstral-2-123B-Instruct-2512 model with chat, streaming, function calling, and reasoning capabilities.

• Tests

  • Integrated new model into nightly benchmark testing and performance evaluation suite.

[coderabbitai]

Warning

Rate limit exceeded

@smfirmin has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 17 minutes and 48 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 17 minutes and 48 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d256ed87-433d-4151-869c-418c17e81dd6

📥 Commits

Reviewing files that changed from the base of the PR and between 537f571 and 34a818c.

📒 Files selected for processing (3)

• .github/workflows/nightly-benchmark.yml
• e2e_test/benchmarks/test_nightly_perf.py
• e2e_test/infra/model_specs.py

📝 Walkthrough

Walkthrough

This PR adds support for mistralai/Devstral-2-123B-Instruct-2512 to the nightly benchmark infrastructure by registering it in the CI workflow, test matrix, and model specifications, with single-worker-only test variant generation.

Changes

Cohort / File(s)	Summary
CI Workflow .github/workflows/nightly-benchmark.yml	Added new model entry mistralai/Devstral-2-123B-Instruct-2512 to the single-worker-h200 job matrix with corresponding slug and test class name TestNightlyDevstral2Single.
Test Infrastructure & Generation e2e_test/benchmarks/test_nightly_perf.py	Registered new nightly model entry for Devstral-2 in _NIGHTLY_MODELS list and introduced _SINGLE_ONLY_NIGHTLY_MODELS set. Modified test class generation logic to produce only "Single" variant (1 worker) for models in this set, while other models continue generating both "Single" and "Multi" variants.
Model Configuration e2e_test/infra/model_specs.py	Added new MODEL_SPECS entry for mistralai/Devstral-2-123B-Instruct-2512 with tensor parallelism (tp: 4), supported features (chat, streaming, function_calling, reasoning), startup timeout (1200 seconds), and backend arguments (--trust-remote-code for both sglang and vllm).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Possibly related PRs

• #327: Introduces refactored single-worker matrix structure that this PR builds upon for new model registration.
• #900: Adds new model entries following the identical pattern across the same three configuration files.
• #795: Implements the same code-level pattern of registering a new model to MODEL_SPECS, _NIGHTLY_MODELS, and the nightly-benchmark workflow.

Suggested labels

ci, tests

Suggested reviewers

• CatherineSue
• key4ng
• slin1237
• XinyueZhang369

Poem

🐰 A model new, with power grand,
Devstral-2 joins the benchmark stand,
Four tensors split, one worker strong,
Through nightly tests it marches along,
Reasoning flows, features glow!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the primary change: adding the Devstral 2 model to H200 nightly benchmarks across all three modified files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request adds the Devstral-2-123B-Instruct-2512 model to the nightly performance benchmarks and infrastructure configurations. It also introduces a mechanism to skip redundant multi-worker test variants for specific models. The reviewer suggests simplifying the test generation logic by checking if _multi_workers > 1 instead of using a hardcoded exclusion set, which would more generically handle models that only require a single worker.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

e2e_test/benchmarks/test_nightly_perf.py (1)

176-188: 🧹 Nitpick | 🔵 Trivial

Clean up _variants from module namespace for consistency.

Line 188 cleans loop temporaries, but _variants is now also a module-level temporary and should be deleted too.

Suggested small cleanup

-del _model_id, _name, _multi_workers, _backends, _extra, _suffix, _count, _cls_name, _cls
+del _model_id, _name, _multi_workers, _backends, _extra, _variants, _suffix, _count, _cls_name, _cls

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@e2e_test/benchmarks/test_nightly_perf.py` around lines 176 - 188, The
module-level temporary _variants is left behind after the loop and should be
removed with the other temporaries; add _variants to the cleanup deletion so the
trailing line becomes del _model_id, _name, _multi_workers, _backends, _extra,
_suffix, _count, _cls_name, _cls, _variants to avoid leaking the variable from
the loop that builds test classes in the block using _NIGHTLY_MODELS and
_make_test_class.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/nightly-benchmark.yml:
- Line 353: The YAML flow-mapping on the line containing the mapping with id:
mistralai/Devstral-2-123B-Instruct-2512, slug:
mistralai-Devstral-2-123B-Instruct-2512, and test_class:
TestNightlyDevstral2Single violates the `braces` rule due to spaces immediately
inside `{ }`; remove the space after `{` and before `}` so the mapping becomes
brace-adjacent (e.g. `{id: ..., slug: ..., test_class: ...}`) to satisfy
yamlint.

---

Outside diff comments:
In `@e2e_test/benchmarks/test_nightly_perf.py`:
- Around line 176-188: The module-level temporary _variants is left behind after
the loop and should be removed with the other temporaries; add _variants to the
cleanup deletion so the trailing line becomes del _model_id, _name,
_multi_workers, _backends, _extra, _suffix, _count, _cls_name, _cls, _variants
to avoid leaking the variable from the loop that builds test classes in the
block using _NIGHTLY_MODELS and _make_test_class.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 419636fa-5ef2-4f14-a65a-ea79ac9fe15d

📥 Commits

Reviewing files that changed from the base of the PR and between 9516b44 and 537f571.

📒 Files selected for processing (3)

• .github/workflows/nightly-benchmark.yml
• e2e_test/benchmarks/test_nightly_perf.py
• e2e_test/infra/model_specs.py

[mergify]

Hi @smfirmin, the DCO sign-off check has failed. All commits must include a Signed-off-by line.

To fix existing commits:

# Sign off the last N commits (replace N with the number of unsigned commits)
git rebase HEAD~N --signoff
git push --force-with-lease

To sign off future commits automatically:

• Use git commit -s every time, or
• VSCode: enable Git: Always Sign Off in Settings
• PyCharm: enable Sign-off commit in the Commit tool window