fix: fix for single-iteration optimizations#209
Merged
andrewklatzke merged 3 commits intoJul 15, 2026
Merged
Conversation
jsonbailey
approved these changes
Jul 15, 2026
andrewklatzke
merged commit Jul 15, 2026
ba5aa63
into
aklatzke/AIC-2263/sdk-dx-improvements
6 checks passed
Merged
andrewklatzke
added a commit
that referenced
this pull request
Jul 17, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-server-sdk-ai: 1.2.0</summary> ## [1.2.0](launchdarkly-server-sdk-ai-1.1.0...launchdarkly-server-sdk-ai-1.2.0) (2026-07-17) ### Features * implements template/raw methods for fetching configs ([1c06d35](1c06d35)) * implements template/raw methods for fetching configs ([#204](#204)) ([5106866](5106866)) </details> <details><summary>ldai_optimizer: 0.2.0</summary> ## [0.2.0](ldai_optimizer-0.1.0...ldai_optimizer-0.2.0) (2026-07-17) ### ⚠ BREAKING CHANGES * Bump minimum LangChain version to 1.0.0 ### Features * ability to specify variation key on config or from_options for optimization package ([c5e39eb](c5e39eb)) * ability to specify variation key on config or from_options for optimization package ([#162](#162)) ([f0c4612](f0c4612)) * add auto-commit option ([4cb8859](4cb8859)) * add optimization for duration ([5d76276](5d76276)) * Add optimization package stub ([#109](#109)) ([ebd5166](ebd5166)) * add shared dataclass for calls so they can be handled by same handler ([31c8385](31c8385)) * add token limit handling ([e8c6692](e8c6692)) * adds ability to optimize for cost ([94de596](94de596)) * adds ability to optimize for cost ([#172](#172)) ([3b4baa3](3b4baa3)) * adds ability to use inverted judges ([0ed243e](0ed243e)) * adds ability to use inverted judges ([#168](#168)) ([23baeb4](23baeb4)) * Adds optimization package stub ([58b7731](58b7731)) * Adds optimization package stub ([cc85a05](cc85a05)) * adds reporting for cost and latency optimization failures ([365fa94](365fa94)) * adds reporting for cost and latency optimization failures ([#180](#180)) ([d267832](d267832)) * all logs -> debug ([2fd55e2](2fd55e2)) * Bump minimum LangChain version to 1.0.0 ([dc592c5](dc592c5)) * Drop support for python 3.9 ([#114](#114)) ([dc592c5](dc592c5)) * dx improvements for optimization package ([7074cfa](7074cfa)) * ground truth optimization path ([44c8c59](44c8c59)) * implement ability to use completions or agents for judge calls ([ea596a7](ea596a7)) * implement additional config optimization api fields ([#196](#196)) ([4ed30d8](4ed30d8)) * implement latency & token tracking for optimizations ([288336e](288336e)) * Implement optimization code paths and functionality for initial release ([#140](#140)) ([5204c47](5204c47)) * implementation of agent optimization + tests ([1712e4f](1712e4f)) * implements LD API client, optimize_from_config path ([2fecd54](2fecd54)) * implements optimize method in SDK, code moved ([9859d08](9859d08)) * partially implement optimize_from_config ([d3e1f96](d3e1f96)) * prevent overfitting via prompt changes and post-processing ([8f9f1e2](8f9f1e2)) * use judge_passed for all calcs ([a8f14de](a8f14de)) ### Bug Fixes * address cursor feedback ([f2f0894](f2f0894)) * adjust iteration logic so validation doesn't consume them ([3042984](3042984)) * better handling of params and custom params for optimization ([572a2aa](572a2aa)) * better handling of params and custom params for optimization ([#163](#163)) ([92f51fa](92f51fa)) * consistency with other makefiles ([b9a5601](b9a5601)) * don't only evaluate final input in GT results ([53f455f](53f455f)) * don't only evaluate final input in GT results ([9bedf9e](9bedf9e)) * ensure cost data is persisted ([4eb0bb0](4eb0bb0)) * fix for single-iteration optimizations ([96eadb7](96eadb7)) * fix for single-iteration optimizations ([#209](#209)) ([ba5aa63](ba5aa63)) * lint ([aee6aa7](aee6aa7)) * lint + missed variable rename ([f8e5509](f8e5509)) * lints + structured output tool rename ([8481690](8481690)) * move failure forward ([3c65a6e](3c65a6e)) * only strip known provider prefixes ([66bc1f0](66bc1f0)) * **optimization:** address Bugbot comments on PR [#140](#140) ([9e57d86](9e57d86)) * **optimization:** fix CI test failures and apply Bugbot-reported fixes ([ffad0cb](ffad0cb)) * **optimization:** match model config by key as well as id for pricing lookup ([a8415be](a8415be)) * **optimization:** remove dead persistence counter code ([c76cb16](c76cb16)) * **optimization:** retry LLM calls on transient provider errors (429/503/529) ([a131e4b](a131e4b)) * **optimization:** use run-chosen context for config judges; guard non-dict JSON ([64e12f2](64e12f2)) * pull model configs if available in options path ([4fc1ecf](4fc1ecf)) * remove unnecessary token path ([dc82818](dc82818)) * sort imports ([c032aaf](c032aaf)) * success path + add test, cursor feedback ([8f3468f](8f3468f)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Mechanical version, changelog, and dependency-floor updates from Release Please; no runtime code changes in this PR. > > **Overview** > **Release Please** version bump only—no application logic changes in this diff. > > **`launchdarkly-server-sdk-ai` 1.2.0** — `pyproject.toml`, `ldai.__version__`, changelog, and provenance docs move from **1.1.0 → 1.2.0**. The new changelog section documents **template/raw config fetch APIs** (shipped in prior commits, e.g. `completion_config_template`). > > **`launchdarkly-ai-optimizer` 0.2.0** — same pattern (**0.1.0 → 0.2.0**) with an expanded **CHANGELOG** (including **LangChain ≥ 1.0** as a breaking note for optimizer consumers). > > **Provider packages** — `launchdarkly-server-sdk-ai-langchain` and `launchdarkly-server-sdk-ai-openai` now require **`launchdarkly-server-sdk-ai>=1.2.0`** instead of **≥1.1.0**. > > **`.release-please-manifest.json`** is updated to match the new package versions. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e86bad4. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
Related issues
https://launchdarkly.atlassian.net/browse/AIC-2980
Describe the solution you've provided
Fixes a bug where ground truth optimizations that had only a single iteration + optimization loop would fail to ever mark the final item as completed. > 1 iterations weren't affected by this bug.
Additionally moves the API call for the first result forward so that failure is evident more immediately.
Describe alternatives you've considered
bug fix
Note
Medium Risk
Changes how optimization results are persisted and how Phase 2 marks non-winner iterations, which affects run completion and auto-commit; limited to the optimization client, not auth or data stores.
Overview
Fixes ground-truth / config-driven runs that never show as completed when Phase 2 (cost/latency) or sparse iterations leave the backend on the wrong result record.
On API persistence,
successpatches now update_last_optimization_result_idso auto-commit’screatedVariationKeylands on the winning iteration instead of the last POSTed Phase 2 row (which could stay RUNNING and dominate run status).Phase 2 behavior changes: model evaluation order is built from
model_choiceswithout re-listing the Phase 1 winner (fallback to the winner only when there are no alternatives); latency/cost gates show “evaluating” placeholders during generation; agent turns use a 120s timeout; and after a successful Phase 2, non-winning iterations are PATCHed as run-level PASSED (success) rather thanfailure, with the true winner updated last so internal winner state and callbacks stay correct.Reviewed by Cursor Bugbot for commit eba89fc. Bugbot is set up for automated code reviews on this repo. Configure here.