Extremely slow to run evaluation in lm-evaluation-harness

[juankost]

I'm trying to run the code on different benchmarks of the lm-evaluation-harness to get a better understanding of the Llamba performance and tradeoffs compared to Llama.

For the Llamba-1B I was trying to run the following benchmarks: MBPP, BBH

CUDA_VISIBLE_DEVICES=0 HF_ALLOW_CODE_EVAL=1 python -m evals.cartesia_lm_eval --model llamba_ssm --model_args pretrained=cartesia-ai/Llamba-1B,trust_remote_code=True,device=cuda:0 --trust_remote_code --confirm_run_unsafe_code --tasks bbh,mbpp --cache_requests true --batch_size auto:8 --output_path llamba_evals/llamba

however, they run extremely slowly, if they even run (i.e. BBH has an estimated completion time of >24h, MPBB seems to be stuck just on the compilation step).

Any help understanding why is it so slow, or how to speed it up , are welcome

I'm using an A100 40GB GPU.