Fix/openvino genai model call by samarth-w · Pull Request #1 · samarth-w/lm-evaluation-harness

samarth-w · 2025-11-03T06:07:17Z

No description provided.

…penVINO pipeline to avoid incompatible arg errors

…uts; prefer logprobs/scores

gemini-code-assist · 2025-11-03T06:07:39Z

Summary of Changes

Hello @samarth-w, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a fundamental incompatibility between the lm_eval framework's evaluation requirements and the current capabilities of the openvino_genai library. It removes the OpenVINOCausalLM integration due to the library's inability to expose per-token log-probabilities, which are crucial for calculating metrics like log-likelihood. A dedicated test script has been introduced to clearly illustrate this technical limitation, providing a clear rationale for the removal of the integration.

Highlights

OpenVINO GenAI Integration Removed: The OpenVINOCausalLM class, which provided the interface for evaluating OpenVINO GenAI models, has been entirely removed from lm_eval/models/optimum_lm_genai.py.
Log-Probability Incompatibility Confirmed: A new diagnostic script (tmp_smoke_test.py) has been added to explicitly demonstrate that the openvino_genai library does not currently provide the necessary per-token log-probability information or full vocabulary distributions required for accurate log-likelihood evaluations within lm_eval.
Discontinuation of Direct Support: This pull request effectively discontinues direct support for OpenVINO GenAI models in lm_eval until the underlying library can provide the granular log-probability data essential for evaluation metrics.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request is a great refactoring of the OpenVINOCausalLM model. By inheriting from HFLM, it significantly reduces code duplication and improves maintainability. The addition of a configurable cache directory is also a welcome improvement.

However, there are a couple of critical issues that need to be addressed:

A Byte Order Mark (BOM) has been added to lm_eval/models/optimum_lm_genai.py, which should be removed.
A new temporary script, tmp_smoke_test.py, has been added with a hardcoded absolute file path, making it non-portable. This file should either be removed or converted into a proper, parameterized test.

Once these issues are resolved, this will be a solid contribution.

gemini-code-assist · 2025-11-03T06:08:25Z

@@ -1,4 +1,4 @@
-import logging
+import logging


A UTF-8 Byte Order Mark (BOM) character (\ufeff) has been introduced at the beginning of this file. This can cause issues with some Python interpreters and tools and is generally not recommended for UTF-8 encoded files. Please remove it.

Suggested change

import logging

import logging

gemini-code-assist · 2025-11-03T06:08:25Z

+
+# --- Configuration ---
+# Define the model path and device for the test.
+MODEL_PATH = r"C:\Users\Administrator\Downloads\openvino.genai\tools\llm_bench\models\gpu_models\google_gemma-2b-it_int4_cw"


The MODEL_PATH is hardcoded to an absolute local path. This makes the script non-portable and will cause it to fail on any other machine. This path should be parameterized, for example, by using command-line arguments or environment variables.

Additionally, temporary or debugging scripts like this one (as suggested by the name tmp_smoke_test.py) should generally not be committed to the repository. Please either convert this into a formal test within the tests directory or remove it from the pull request.

Copilot

Pull Request Overview

This PR removes a large block of duplicated code from the OpenVINO GenAI model implementation and adds configuration for model caching. The purpose is to clean up the codebase by eliminating redundant class definitions and methods while improving the cache directory handling.

Key changes:

Removed ~570 lines of duplicate code including a complete redundant class definition
Added cache_dir parameter support for OpenVINO model caching
Added a temporary smoke test file for debugging OpenVINO GenAI functionality

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tmp_smoke_test.py	New temporary test file for analyzing OpenVINO GenAI log probability functionality
lm_eval/models/optimum_lm_genai.py	Removed duplicate code and added cache_dir parameter support for model caching

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-03T07:43:11Z

+
+# --- Setup ---
+# Configure logging for clear, professional output.
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+# --- Configuration ---
+# Define the model path and device for the test.
+MODEL_PATH = r"C:\Users\Administrator\Downloads\openvino.genai\tools\llm_bench\models\gpu_models\google_gemma-2b-it_int4_cw"


Hard-coded absolute path should be made configurable or use environment variables to avoid system-specific dependencies in test code.

Suggested change

# --- Setup ---

# Configure logging for clear, professional output.

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# --- Configuration ---

# Define the model path and device for the test.

MODEL_PATH = r"C:\Users\Administrator\Downloads\openvino.genai\tools\llm_bench\models\gpu_models\google_gemma-2b-it_int4_cw"

import os

# --- Setup ---

# Configure logging for clear, professional output.

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# --- Configuration ---

# Define the model path and device for the test.

MODEL_PATH = os.environ.get(

"MODEL_PATH",

r"C:\Users\Administrator\Downloads\openvino.genai\tools\llm_bench\models\gpu_models\google_gemma-2b-it_int4_cw"

)

if "MODEL_PATH" not in os.environ:

logging.warning("MODEL_PATH environment variable not set. Using default model path: %s", MODEL_PATH)

Copilot · 2025-11-03T07:43:11Z

+    logging.info("="*70)
+
+    full_text = prompt + choice
+    raw_tokens = tokenizer.encode(full_text).input_ids.data.tolist()[0]


Potential IndexError if the tokenizer returns an empty list. Consider adding bounds checking before accessing index [0].

Suggested change

raw_tokens = tokenizer.encode(full_text).input_ids.data.tolist()[0]

tokenized_list = tokenizer.encode(full_text).input_ids.data.tolist()

if not tokenized_list:

logging.error("Tokenizer returned an empty list for input: '%s'. Cannot proceed with log-likelihood analysis.", full_text)

return

raw_tokens = tokenized_list[0]

Copilot · 2025-11-03T07:43:11Z

+
+    formatted_prompt = f"{prompt}\n" + "\n".join([f"{chr(65+i)}) {choice}" for i, choice in enumerate(choices)]) + "\nAnswer:"
+
+    raw_tokens = tokenizer.encode(formatted_prompt).input_ids.data.tolist()[0]


Potential IndexError if the tokenizer returns an empty list. Consider adding bounds checking before accessing index [0].

Suggested change

raw_tokens = tokenizer.encode(formatted_prompt).input_ids.data.tolist()[0]

token_list = tokenizer.encode(formatted_prompt).input_ids.data.tolist()

if not token_list:

logging.error("Tokenization failed: the tokenizer returned an empty list for the prompt.")

return

raw_tokens = token_list[0]

samarth-w added 2 commits October 31, 2025 18:29

fix(optimum_lm_genai): consolidate aliases and pass decoded text to O…

1bff780

…penVINO pipeline to avoid incompatible arg errors

fix(optimum_lm_genai): robust _model_call using TokenizedInputs/timeo…

9be5122

…uts; prefer logprobs/scores

samarth-w merged commit f32dbd6 into main Nov 3, 2025
4 of 5 checks passed

gemini-code-assist Bot reviewed Nov 3, 2025

View reviewed changes

samarth-w requested a review from Copilot November 3, 2025 07:42

Copilot AI reviewed Nov 3, 2025

View reviewed changes

Navigation Menu

Fix/openvino genai model call - #1

Fix/openvino genai model call#1
samarth-w merged 2 commits into
mainfrom
fix/openvino-genai-model-call

samarth-w commented Nov 3, 2025

Uh oh!

Uh oh!

gemini-code-assist Bot commented Nov 3, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Nov 3, 2025

Uh oh!

gemini-code-assist Bot Nov 3, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 3, 2025

Uh oh!

Copilot AI Nov 3, 2025

Uh oh!

Copilot AI Nov 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    raw_tokens = tokenizer.encode(full_text).input_ids.data.tolist()[0]
+    tokenized_list = tokenizer.encode(full_text).input_ids.data.tolist()
+    if not tokenized_list:
+        logging.error("Tokenizer returned an empty list for input: '%s'. Cannot proceed with log-likelihood analysis.", full_text)
+        return
+    raw_tokens = tokenized_list[0]


		formatted_prompt = f"{prompt}\n" + "\n".join([f"{chr(65+i)}) {choice}" for i, choice in enumerate(choices)]) + "\nAnswer:"

		raw_tokens = tokenizer.encode(formatted_prompt).input_ids.data.tolist()[0]

-    raw_tokens = tokenizer.encode(formatted_prompt).input_ids.data.tolist()[0]
+    token_list = tokenizer.encode(formatted_prompt).input_ids.data.tolist()
+    if not token_list:
+        logging.error("Tokenization failed: the tokenizer returned an empty list for the prompt.")
+        return
+    raw_tokens = token_list[0]

Conversation

samarth-w commented Nov 3, 2025

Uh oh!

Uh oh!

gemini-code-assist Bot commented Nov 3, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants