This project investigates emotional bias in large language models (LLMs) across multiple languages.
It evaluates LLM outputs’ emotional tone and toxicity using the Detoxify model and explores possible bias mitigation strategies.
| Component | Description |
|---|---|
| Goal | Detect and analyze emotional bias in LLM-generated text accrossing different languages |
| Models | Qwen2.5, LLaMA-3, or other open LLMs (via vLLM) |
| Evaluator | Detoxify (for toxicity & emotion scoring) |
| Dataset | XNLI multilingual benchmark，Toxigen |
| Frameworks | vLLM, datasets |
| Environment | Singularity container with GPU support |
| Model | url |
|---|---|
| Qwen-2.5-1.5B-Instruct | https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct |
| Qwen-2.5-3B-Instruct | https://huggingface.co/Qwen/Qwen2.5-3B-Instruct |
| Qwen-2.5-7B-Instruct | http://huggingface.co/Qwen/Qwen2.5-7B-Instruct |
| LLama-3.2-1B-Instruct | https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct |
| LLama-3.2-3B-Instruct | https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct |
| LLama-3.1-8B-Instruct | https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct |
| Phase | Description | Status | Notes |
|---|---|---|---|
| 1️⃣ Demo 1 | Environment setup and benchmark completed | ✅ Completed | benchmark.py |
| 2️⃣ Demo 2 | Experiment configuration and environment replication | ✅ Completed | evl_[model_name].py |
| 3️⃣ Demo 3 | Try some possible way to mitigate the bias | ✅ Completed | prompts etc. |
| 4️⃣ Report | Analysis, visualization, and report writing | 🚧 Working | Summarize results, discuss and reporting |
This container includes the vLLM framework for LLM deployment, along with essential dependencies such as datasets, pandas, and torch.
singularity pull vllm_base.sif docker://tiyamo/vllm_baseInstall Detoxify , a transformer-based model used for emotional and toxicity analysis
pip install detoxifyThis project is MIT Licensed. See LICENSE for details.