RBLU

Reverse Bilingual Language Understanding (RBLU): A benchmark to evaluate the reverse inference ability of large language models

The reverse inference process of RBLU benchmark

Input the question to get the answer, and then input the answer to get the question. Finally, calculate the similarity of the input question and the output question
Explore the doc »

View Demo · Report Bug · Request Feature

Content

• Built With
• About The Project
  • Dependencies
  • Installation
• Project Tree
• Usage
• Roadmap
• Author
• Acknowledgments

Build With

• 
• 
• 
• 

Getting Start

Dependencies

• accelerate = "^0.34.2"
• datasets = "^3.0.0"
• evaluate = "^0.4.3"
• matplotlib = "^3.9.2"
• pandas = "^2.2.2"
• python = "^3.12"
• rouge-chinese = "^1.0.3"
• rouge-score = "^0.1.2"
• transformers = "^4.44.2"
• sentence-transformers = "^3.1.0"
• jieba = "^0.42.1"
• wandb = "^0.18.0"
• tiktoken = "^0.7.0"
• pyecharts = "^2.0.6"
• seaborn = "^0.13.2"
• plotly = "^5.24.1"

Installation

1. install poetry
2. Clone the repo
3. Installing dependencies with poetry

poetry install

Project Tree

eg:

RBLU 
├── LICENSE.txt
├── README.md
├── /figure/
├── /src/
│  ├── /chart/
│  ├── /data/
│  ├── /result/
│  ├── /score/
│  ├── /rblu/
│  │  ├── config.yaml
│  │  └── main.py
│  │  └── dataload.py
│  │  └── evaluation.py
│  │  └── metric.py
│  │  └── process.py
│  │  └── path.py
│  │  └── proxy.py
└── /tests/

Usage

You can find the result for our paper in result, score, chart. You can re-run the project by the following command：

poetry run eval

Or re-draw the chart by:

poetry run draw --suffix png

Roadmap

We tested three open source models, LLAMA3.1-8BInstruct, GLM4-9B-Chat, and Qwen2-7B-Instruct on the RBLU benchmark with ROUGE and BERT-Score, the results are as follows:

Language	Domain	Model Name	Rouge1	Rouge2	RougeL	RougeLsum	BERT score
English	Financial	GLM4	0.1322	0.0251	0.0929	0.1047	0.5161
		LLAMA3.1	0.1200	0.0212	0.0817	0.0970	0.4632
		Qwen2	0.1260	0.0217	0.0878	0.1005	0.5181
	Legal	GLM4	0.1409	0.0320	0.0930	0.1052	0.5194
		LLAMA3.1	0.1271	0.0258	0.0827	0.0978	0.4686
		Qwen2	0.1255	0.0203	0.0814	0.0964	0.4792
	Medical	GLM4	0.2115	0.0718	0.1588	0.1600	0.5799
		LLAMA3.1	0.1865	0.0652	0.1428	0.1452	0.5512
		Qwen2	0.1962	0.0740	0.1509	0.1509	0.5600
Chinese	Financial	GLM4	0.2131	0.0604	0.2120	0.2121	0.7878
		LLAMA3.1	0.1566	0.0482	0.1555	0.1551	0.7386
		Qwen2	0.1210	0.0295	0.1218	0.1215	0.7398
	Legal	GLM4	0.0587	0.0108	0.0587	0.0584	0.7090
		LLAMA3.1	0.0355	0.0109	0.0349	0.0364	0.6527
		Qwen2	0.0605	0.0088	0.0612	0.0615	0.6957
	Medical	GLM4	0.0893	0.0214	0.0880	0.0890	0.5723
		LLAMA3.1	0.0540	0.0111	0.0552	0.0542	0.5390
		Qwen2	0.0843	0.0195	0.0841	0.0843	0.6198

Similarity Scores in Multi-rounds

The right side indicates 3 domains, and the top side indicates 2 score types and 2 languages. "Cosine" represents "BERT-Score", the cosine similarity of vectorized answer texts, while "Rouge1" is the corresponding Rouge-1 score. The datasets are in English and Chinese. In the legend, "Original" indicates Score_Original, and "Previous" indicates Score_Previous. The x-axis of each subplot shows the number of rounds (1–4), and the y-axis shows similarity scores (0.0–1.0).

Questions Answers

Some insights

• GLM4 Performance: GLM4 demonstrates the strongest reverse inference performance among the models tested.

• Semantic vs. Syntactic: LLMs generally capture semantic meaning more effectively than syntactic structure.

• Cognitive Inertia: LLMs exhibit cognitive inertia, as they tend to generate increasingly similar questions over multiple rounds.

• Forward vs. Reverse Inference: LLMs show stronger forward inference capabilities than reverse inference.

• Language Differences:

  • Chinese: Due to the flexibility of expressions and varied word choices in Chinese, the outputs display greater semantic flexibility.
  • English: English outputs maintain higher syntactic consistency due to the stricter syntactic rules in the language.

Author

Haowei Wang

AlexLiu

License

This project is licensed under the MIT License. For more details, please refer to LICENSE.txt

Acknowledgments

• GitHub Pages
• isort
• ruff