feat: expose ONNX Runtime session config entries through InitOptions - #292
Merged
Anush008 merged 2 commits intoSep 12, 2026
Merged
Conversation
fastembed builds the ONNX Runtime SessionBuilder internally. Callers had no way to set session-only configuration entries such as mlas.disable_kleidiai, which controls a per-session GEMM buffer switch on Apple SME hardware. Add a session_config field to every init-options struct. Add a with_session_config(key, value) builder method that appends one entry. init_session_builder applies each entry with SessionBuilder::with_config_entry after setting intra-op threads and before the DirectML block. Update all 10 call sites that build a session from init options. All From impls between default and user-defined option structs now carry session_config across.
Anush008
reviewed
Sep 12, 2026
Owner
|
Thanks for the contribution @paulnsorensen. |
github-actions Bot
pushed a commit
that referenced
this pull request
Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12) ### 🍕 Features * expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9)) * inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d)) ### 📝 Documentation * Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
|
🎉 This PR is included in version 6.1.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Anush008
pushed a commit
that referenced
this pull request
Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12) ### 🍕 Features * expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9)) * inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d)) ### 📝 Documentation * Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
Anush008
pushed a commit
that referenced
this pull request
Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12) ### 🍕 Features * expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9)) * inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d)) ### 📝 Documentation * Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
Anush008
pushed a commit
that referenced
this pull request
Sep 12, 2026
## [6.1.0](v6.0.3...v6.1.0) (2026-09-12) ### 🍕 Features * expose ONNX Runtime session config entries through InitOptions ([#292](#292)) ([adfefb9](adfefb9)) * inference-free SPLADE (opensearch-neural-sparse-encoding-doc-v3-gte) ([#289](#289)) ([6b9e45d](6b9e45d)) ### 📝 Documentation * Update LICENSE ([#290](#290)) ([a535a2f](a535a2f))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #291.
What changed
init_session_buildertakes asession_config: Vec<(String, String)>parameter. It applies each entry withSessionBuilder::with_config_entryafterwith_intra_threads. The DirectML block is unchanged.InitOptions<M>andInitOptionsWithLength<M>gain asession_configfield and awith_session_config(key, value)builder method. The default is empty.InitOptionsUserDefined,RerankInitOptionsUserDefined, andImageInitOptionsUserDefinedgain the same field and method. TheFromimpls copy the field.init_session_buildercall sites pass the field through.with_intra_threadsexample.Why
Some ONNX Runtime behaviors have a per-session switch only.
mlas.disable_kleidiaiis one example. On Apple Silicon with SME, ONNX Runtime 1.28.0 keeps packed GEMM buffers in thread-local storage. The buffers never shrink. A long-lived process retains gigabytes after one embedding pass. Withwith_session_config("mlas.disable_kleidiai", "1")the same process retains 216 MB instead of 2189 MB. See #291 and microsoft/onnxruntime#29538.The shape follows the existing
execution_providersandintra_threadspassthroughs.Usage
Tests
init_session_builder_applies_config_entrycalls the realSessionBuilderwith one entry and asserts success. It loads no model.with_session_configappends in order and that theFromimpls carry the field.cargo fmt --all -- --check,cargo clippy ... -- -D warnings, andcargo test --libpass withort-download-binaries-native-tls,hf-hub-native-tls.Compatibility
The option structs are
#[non_exhaustive], so the new field is not a breaking change for callers that use the builder methods. Callers that construct the structs by field name inside this crate are updated.🤖 Generated with Claude Code