Reclaim cached GenAI CUDA resources after the last CUDA model unloads

[aciddelgado]

Summary

Address the post-unload memory retention reported in #1079 by invoking GenAI’s device-resource release API after Foundry unloads its last tracked CUDA model.

Destroying a model alone can leave several GiB retained by GenAI’s long-lived dummy session/allocator. This change releases those cached resources without requiring manager shutdown, unloading the CUDA add-on, or unregistering the execution provider.

Changes

• Record each loaded model’s actual GenAI device type.
• Call  OgaReleaseDeviceResources("CUDA")  after model destruction only when no remaining tracked model uses CUDA.
• Preserve the existing guard against unloading models with live sessions.
• Add optional symbol lookup for already-loaded libraries on Windows and POSIX, without changing library lifetime or introducing a hard import dependency.
• Skip resource release when the API is unavailable.
• Log release failures as warnings while preserving successful model unload.

Compatibility

Automatic resource release is limited to CUDA. Optional lookup allows otherwise-compatible GenAI binaries without the new export to continue unloading models normally; other GenAI API/version requirements still apply.

Related issue: #1079

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 15, 2026 10:16pm UTC

[Copilot]

🟡 Changes recommended

Stable release dependencies lack the cleanup symbol, making the fix a silent no-op for published builds.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds conditional GenAI CUDA resource cleanup after the final CUDA model unload.

Changes:

• Tracks each model’s GenAI device type.
• Dynamically resolves and invokes device-resource cleanup.
• Updates native dependency discovery and WinML package sourcing.

File summaries

File	Description
sdk_v2/python/src/foundry_local_sdk/_native/lib_loader.py	Prefers colocated ORT/GenAI binaries.
sdk_v2/cpp/src/platform/windows/dynlib_loader.cc	Adds Windows symbol lookup.
sdk_v2/cpp/src/platform/posix/dynlib_loader.cc	Adds POSIX symbol lookup.
sdk_v2/cpp/src/platform/dynlib_loader.h	Declares symbol-resolution API.
sdk_v2/cpp/src/inferencing/model_load_manager.cc	Releases CUDA resources after final unload.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h	Stores model device type.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc	Reads the GenAI device type.
sdk_v2/cpp/cmake/FindWinMLEpCatalog.cmake	Switches the WinML fallback feed.

Review details

Suppressed comments (1)

sdk_v2/cpp/src/inferencing/model_load_manager.cc:327

• The new last-device gate and warning-only release behavior have no automated coverage. ModelLoadManagerUnloadTest currently exercises only a CPU model, so regressions such as releasing while another CUDA model remains, failing to release the final one, or propagating symbol/release failures would be undetected. Please add an injectable release callback/symbol resolver and cover these branches.

  const bool device_still_in_use =
      std::any_of(loaded_models_.begin(), loaded_models_.end(),
                  [&device_type](const auto& entry) { return entry.second->DeviceType() == device_type; });
  if (device_type == "CUDA" && !device_still_in_use) {
    ReleaseDeviceResourcesIfAvailable(device_type, logger_);

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.