Request for Publishing LLM Responses

[subhendukhatuya]

I am working on training an LLM router and noticed that most routing approaches (e.g MIRT-BERT) require LLM responses on the training dataset to learn routing decisions before inference.

Would it be possible to release the responses generated by the candidate LLMs for the dataset used in the paper?

[jiarong0907]

Hi @subhendukhatuya, Thanks for checking out RouterArena.

To clarify, RouterArena is an evaluation platform, and all queries in it are used as testing dataset. Thus, it is a bit unreasonable to use the RouterArena dataset for training. In the paper, for all routers, we follow their training datasets and methods to train them.

Are there any specific routers you found that lack training data? We are happy to clarify more.

[subhendukhatuya]

Yes, I understand that RouterArena is meant purely as an evaluation platform and that the queries are used as a test dataset rather than for training.

My question is mainly about aligning the candidate LLM sets. Each baseline router in the paper is trained using responses from a specific set of candidate LLMs, but RouterArena evaluates routing decisions over a different set of candidate LLMs.

For example, the MIRT-Router (Table 10 in the paper: https://arxiv.org/pdf/2506.01048) is trained with a particular list of LLMs. However, RouterArena defines its own candidate model list here:
https://github.com/RouteWorks/RouterArena/blob/main/model_cost/model_cost.json

If we want to retrain or adapt MIRT-Router so that it can make routing decisions over the RouterArena candidate model set, it seems that we would need response data from those additional LLMs as well.

So my question is: is there any available response dataset for the RouterArena candidate models, or is there a recommended way to retrain baseline routers (like MIRT-Router) so that they can operate over the RouterArena model list?

[jiarong0907]

Hi @subhendukhatuya, to clarify a bit, as a leaderboard, we don't restrict on the model pool a router can use. We believe it is part of the router design. In other words, we are not comparing routing algorithms, but the final router as a whole.

Thus, the model list here is extensible based on your router requirement.

If one wants to adapt MIRT-Router for latest models, he has to evaluate the query using the desired model. We don't have the cached inference results for that. And we don't restrict ones from using any training queries for their router.

We will release a new training dataset with more recent models for router training in a month or so. Stay tuned.

[subhendukhatuya]

Thanks @jiarong0907 for the reponse. So how I can reproduce the leaderboard present in routerarena?

[jiarong0907]

Which router you cannot reproduce?

[subhendukhatuya]

Hi @subhendukhatuya, Thanks for checking out RouterArena.

To clarify, RouterArena is an evaluation platform, and all queries in it are used as testing dataset. Thus, it is a bit unreasonable to use the RouterArena dataset for training. In the paper, for all routers, we follow their training datasets and methods to train them.

Are there any specific routers you found that lack training data? We are happy to clarify more.

Just a clarifying question--When evaluating baseline routers, I assume that responses from the set of candidate LLMs ( Table 5 in https://arxiv.org/pdf/2510.00202 ) were first generated to construct the training data required for the baseline router
How do you ensure that the training samples are not included in the evaluation set? For example, MIRT-Router uses some samples from MMLU and MATH as training data — what measures are taken to prevent overlap between training and evaluation samples?

[jiarong0907]

@subhendukhatuya We have no good way to prevent any LLM from doing that. If these models are trained professionally, they should only use the benchmarks training set, rather than the testset.