OpenAI API script missing

[IlyaGusev]

Hey! A script called "scripts/run_benchmark_openai_api.sh" is mentioned in the README, but it does not exist.

Can you also provide a base_url parameter there?

[artemorloff]

hi! openai script has been removed. We will update readme file and provide examples of running openai models via the existing scripts

[artemorloff]

one more time thank you for so much paying attention to this problem. We have updated MODEL SCORING file with the newest examples of running MERA benchmark with HF, vLLM, OpenAI. We also provided the code to run HF models via servers that imitate OpenAI API. Hope, now you would be able to run any model you want

[artemorloff]

If you have any questions concerning running MERA on lm-evaluation-harness, do not hesitate to ask. Further details on running models may also be found in long-standing and regularly updated documentation: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs