Skip to content

feat: expose ONNX Runtime session config entries through InitOptions - #292

Merged
Anush008 merged 2 commits into
Anush008:mainfrom
paulnsorensen:feat/session-config-entries
Sep 12, 2026
Merged

feat: expose ONNX Runtime session config entries through InitOptions#292
Anush008 merged 2 commits into
Anush008:mainfrom
paulnsorensen:feat/session-config-entries

Conversation

@paulnsorensen

Copy link
Copy Markdown
Contributor

Closes #291.

What changed

  • init_session_builder takes a session_config: Vec<(String, String)> parameter. It applies each entry with SessionBuilder::with_config_entry after with_intra_threads. The DirectML block is unchanged.
  • InitOptions<M> and InitOptionsWithLength<M> gain a session_config field and a with_session_config(key, value) builder method. The default is empty.
  • InitOptionsUserDefined, RerankInitOptionsUserDefined, and ImageInitOptionsUserDefined gain the same field and method. The From impls copy the field.
  • All ten init_session_builder call sites pass the field through.
  • README: one line next to the with_intra_threads example.

Why

Some ONNX Runtime behaviors have a per-session switch only. mlas.disable_kleidiai is one example. On Apple Silicon with SME, ONNX Runtime 1.28.0 keeps packed GEMM buffers in thread-local storage. The buffers never shrink. A long-lived process retains gigabytes after one embedding pass. With with_session_config("mlas.disable_kleidiai", "1") the same process retains 216 MB instead of 2189 MB. See #291 and microsoft/onnxruntime#29538.

The shape follows the existing execution_providers and intra_threads passthroughs.

Usage

let model = TextEmbedding::try_new(
    TextInitOptions::new(EmbeddingModel::SnowflakeArcticEmbedS)
        .with_session_config("mlas.disable_kleidiai", "1"),
)?;

Tests

  • init_session_builder_applies_config_entry calls the real SessionBuilder with one entry and asserts success. It loads no model.
  • Unit tests assert that with_session_config appends in order and that the From impls carry the field.
  • cargo fmt --all -- --check, cargo clippy ... -- -D warnings, and cargo test --lib pass with ort-download-binaries-native-tls,hf-hub-native-tls.

Compatibility

The option structs are #[non_exhaustive], so the new field is not a breaking change for callers that use the builder methods. Callers that construct the structs by field name inside this crate are updated.

🤖 Generated with Claude Code

fastembed builds the ONNX Runtime SessionBuilder internally. Callers
had no way to set session-only configuration entries such as
mlas.disable_kleidiai, which controls a per-session GEMM buffer switch
on Apple SME hardware.

Add a session_config field to every init-options struct. Add a
with_session_config(key, value) builder method that appends one entry.
init_session_builder applies each entry with
SessionBuilder::with_config_entry after setting intra-op threads and
before the DirectML block.

Update all 10 call sites that build a session from init options. All
From impls between default and user-defined option structs now carry
session_config across.
Comment thread README.md Outdated
@Anush008

Copy link
Copy Markdown
Owner

Thanks for the contribution @paulnsorensen.

@Anush008
Anush008 merged commit adfefb9 into Anush008:main Sep 12, 2026
github-actions Bot pushed a commit that referenced this pull request Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12)

### 🍕 Features

* expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9))
* inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d))

### 📝 Documentation

* Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 6.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Anush008 pushed a commit that referenced this pull request Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12)

### 🍕 Features

* expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9))
* inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d))

### 📝 Documentation

* Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
Anush008 pushed a commit that referenced this pull request Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12)

### 🍕 Features

* expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9))
* inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d))

### 📝 Documentation

* Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
Anush008 pushed a commit that referenced this pull request Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12)

### 🍕 Features

* expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9))
* inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d))

### 📝 Documentation

* Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose ONNX Runtime session configuration entries through InitOptions

2 participants