feat: support stable diffusion v1-5 with openvino by ziyuanguo1998 · Pull Request #282 · microsoft/olive-recipes

ziyuanguo1998 · 2026-04-01T03:00:26Z

No description provided.

Copilot

Pull request overview

This PR adds an AITK workflow/recipe to optimize and evaluate Stable Diffusion v1-5 using OpenVINO (Intel CPU/GPU), including new Olive pass configuration for OpenVINO IO updates and encapsulation, plus model registry/runtime updates.

Changes:

Add an OpenVINO workflow (sd_ov_workflow.py + JSON + UI .json.config) and register it in the model project metadata.
Extend per-submodel Olive configs to include OpenVINOIoUpdate + OpenVINOEncapsulation and adjust UNet example inputs for static shapes.
Update OpenVINO saving logic to emit a Diffusers ONNX pipeline layout, and update the global model list to include Intel CPU/GPU runtimes for SD v1-5.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sd-legacy-stable-diffusion-v1-5/aitk/user_script.py	Adjust UNet OpenVINO example inputs to batch=1 and float32 timestep for static-shape workflows.
sd-legacy-stable-diffusion-v1-5/aitk/stable_diffusion.py	Pass the PyTorch pipeline into the OpenVINO “save model info” step.
sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py	Major OpenVINO path changes: config rewriting, submodel artifact handling, and pipeline saving.
sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/onnx_patch.py	Add `is_ov_save` mode to copy OV/ONNX artifacts during `save_pretrained`.
sd-legacy-stable-diffusion-v1-5/aitk/sd_ov_workflow.py	New workflow entrypoint to run conversion/optimization and (re)use the existing evaluator path.
sd-legacy-stable-diffusion-v1-5/aitk/sd_ov_workflow.json.config	New AITK UI config for OpenVINO workflow with CPU/GPU runtime selection.
sd-legacy-stable-diffusion-v1-5/aitk/sd_ov_workflow.json	New Olive workflow wrapper config pointing at the AITK python workflow script.
sd-legacy-stable-diffusion-v1-5/aitk/model_project.config	Register `sd_ov_workflow.json` as a template in the project.
sd-legacy-stable-diffusion-v1-5/aitk/info.yml	Add OpenVINO recipe metadata for cpu/gpu devices.
sd-legacy-stable-diffusion-v1-5/aitk/config_vae_encoder.json	Add OpenVINO IO update + encapsulation passes for VAE encoder.
sd-legacy-stable-diffusion-v1-5/aitk/config_vae_decoder.json	Add OpenVINO IO update + encapsulation passes for VAE decoder.
sd-legacy-stable-diffusion-v1-5/aitk/config_unet.json	Add OpenVINO IO update + encapsulation passes for UNet.
sd-legacy-stable-diffusion-v1-5/aitk/config_text_encoder.json	Add OpenVINO IO update + encapsulation passes for text encoder.
sd-legacy-stable-diffusion-v1-5/aitk/config_safety_checker.json	Add OpenVINO IO update + encapsulation passes for safety checker (even though OpenVINO path disables it).
.aitk/requirements/requirements-IntelNPU-SD.txt	Add SD-specific requirements set for Intel NPU environments.
.aitk/configs/model_list.json	Add IntelCPU/IntelGPU runtimes for SD v1-5 in the model registry.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-01T03:07:59Z

sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py

+
+    worker_script = os.path.abspath('winml.py')
+    result = subprocess.check_output([sys.executable, worker_script], text=True)
+    paths = json.loads(result)


register_execution_providers() uses os.path.abspath(...), but os is not imported in this module anymore. This will raise a NameError the first time the function is called. Add import os at the top (or import it inside the function) to make this callable.

Copilot · 2026-04-01T03:07:59Z

sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py

+    # model_info_path = optimized_model_dir / OV_OPTIMIZED_MODEL_INFO
+    # with model_info_path.open("w") as model_info_file:
+    #     json.dump(model_info, model_info_file, indent=4)


save_ov_model_info() no longer writes ov_optimized_model_info.json, but get_ov_pipeline() still unconditionally opens that file. As a result, OpenVINO inference via stable_diffusion.py --provider openvino will fail with FileNotFoundError. Either restore writing this mapping file (and ensure it matches the saved artifact layout) or update get_ov_pipeline()/the OpenVINO loading path to use the new save_pretrained directory structure instead of OV_OPTIMIZED_MODEL_INFO.

Suggested change

# model_info_path = optimized_model_dir / OV_OPTIMIZED_MODEL_INFO

# with model_info_path.open("w") as model_info_file:

# json.dump(model_info, model_info_file, indent=4)

model_info_path = optimized_model_dir / OV_OPTIMIZED_MODEL_INFO

with model_info_path.open("w") as model_info_file:

json.dump(model_info, model_info_file, indent=4)

Copilot · 2026-04-01T03:07:59Z

sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py

+    # optimized_model_path = optimized_model_dir / submodel
+    # shutil.copytree(output_model_dir, optimized_model_path)
+    # model_path = (optimized_model_path / submodel).with_suffix(".xml")
+    folder = Path(output_model_dir)
+    onnx_files = sorted(folder.glob("*.onnx"))
+    if onnx_files:
+        first_file = onnx_files[0]
+        target = folder / "model.onnx"
+        first_file.rename(target)
+        print(f"Renamed {first_file.name} -> {target.name}")
+    optimized_model_path_map[submodel] = str(output_model_dir)
+
+


save_optimized_ov_submodel() renames the first *.onnx file in the Olive candidate directory to model.onnx. This mutates the Olive cache/output in-place and can fail if model.onnx already exists or if the “first” ONNX file isn’t the correct one. Prefer copying (or selecting a deterministic/expected filename from the candidate metadata) into the final output structure rather than renaming in the cache directory.

Suggested change

# optimized_model_path = optimized_model_dir / submodel

# shutil.copytree(output_model_dir, optimized_model_path)

# model_path = (optimized_model_path / submodel).with_suffix(".xml")

folder = Path(output_model_dir)

onnx_files = sorted(folder.glob("*.onnx"))

if onnx_files:

first_file = onnx_files[0]

target = folder / "model.onnx"

first_file.rename(target)

print(f"Renamed {first_file.name} -> {target.name}")

optimized_model_path_map[submodel] = str(output_model_dir)

optimized_model_path = optimized_model_dir / submodel

# Copy the entire candidate directory into the final optimized model directory

shutil.copytree(output_model_dir, optimized_model_path, dirs_exist_ok=True)

# Within the copied directory, ensure there is a deterministic 'model.onnx' if possible

folder = Path(optimized_model_path)

onnx_files = sorted(folder.glob("*.onnx"))

if onnx_files:

first_file = onnx_files[0]

target = folder / "model.onnx"

if not target.exists():

shutil.copy2(first_file, target)

print(f"Copied {first_file.name} -> {target.name}")

else:

print(f"Target {target.name} already exists; leaving existing file unchanged.")

optimized_model_path_map[submodel] = str(optimized_model_path)

Copilot · 2026-04-01T03:07:59Z

sd-legacy-stable-diffusion-v1-5/aitk/sd_utils/ov.py

+        vae_decoder,
+        text_encoder,
+        tokenizer,
+        unet,
        scheduler: Union[DDIMScheduler, PNDMScheduler, LMSDiscreteScheduler],


OVStableDiffusionPipeline.__init__ removed the OpenVINO Model type annotations, but the docstring still documents these parameters as Model. Either update the docstring types to match the new expectations (e.g., compiled model/infer request types) or reintroduce the type hints via typing.TYPE_CHECKING imports to keep the API documentation accurate.

xieofxie · 2026-04-01T03:08:21Z

sd-legacy-stable-diffusion-v1-5/aitk/sd_ov_workflow.json.config

+        "vae-decoder-latency-avg": "vae-decoder-latency-avg"
+    },
+    "evalNoDataConfig": true,
+    "memoryGbSuggested": 30,


maybe remove this

xieofxie · 2026-04-01T03:09:49Z

sd-legacy-stable-diffusion-v1-5/aitk/config_unet.json

+            "static": true,
+            "input_shapes": [
+                [ 1, 4, 64, 64 ],
+                [ ],


we need to update sample to consider both cases

xieofxie · 2026-04-01T03:47:26Z

sd-legacy-stable-diffusion-v1-5/aitk/info.yml

 aitk:
  modelInfo:
    id: "huggingface/stable-diffusion-v1-5/stable-diffusion-v1-5"
    version: 1


update version to 2

ZiyuanGuo added 3 commits April 1, 2026 04:22

support ov convert

93d22c6

fix bug

fc43f99

fix convert

f28665d

ziyuanguo1998 requested review from a team as code owners April 1, 2026 03:00

Copilot AI review requested due to automatic review settings April 1, 2026 03:00

Copilot started reviewing on behalf of ziyuanguo1998 April 1, 2026 03:01 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

xieofxie reviewed Apr 1, 2026

View reviewed changes

fix pre-commit

8c0ce95

xieofxie reviewed Apr 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support stable diffusion v1-5 with openvino#282

feat: support stable diffusion v1-5 with openvino#282
ziyuanguo1998 wants to merge 4 commits intomainfrom
ziyuan/sd-ov

ziyuanguo1998 commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 1, 2026

Uh oh!

Copilot AI Apr 1, 2026

Uh oh!

Copilot AI Apr 1, 2026

Uh oh!

Copilot AI Apr 1, 2026

Uh oh!

xieofxie Apr 1, 2026

Uh oh!

xieofxie Apr 1, 2026

Uh oh!

xieofxie Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ziyuanguo1998 commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

xieofxie Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

xieofxie Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

xieofxie Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants