Repository navigation
fix(decision): decide through experimental_decide and reject unsupported models at config load - #916
Merged
Merged
Conversation
… at config load - peer ai ^7.0.128: first release with experimental_decide (7.0.127 lacks it) - accept decision models and deprecated evaluation models; types from ai - drop the @ai-sdk/provider pin that duplicated ai's copy - unsupported model is INVALID_CONFIG naming the fix, not a TypeError - missing API key names the env var the provider reads - docs, README, skill use decisionModel()
Read experimental_decide off the ai namespace so an older ai loads the package and decisionExecutor() throws INVALID_CONFIG naming 7.0.128, instead of a module link error. An uncalled provider factory as model no longer prints its source.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Re-trigger cubic
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.17.0 · 1m 50s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed33 agent steps · 26 replayed from cache · 18 model calls · 71.0k tokens (93% cached) · $0.0031 All 33 tests in 5 files
e2e 0.17.0 · 2m 46s · web · run artifacts |
Contributor
🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped25 agent steps · 143 model calls · 1.0M tokens (79% cached) · $0.0756
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped | 25 steps · 143 calls | 15m 25s | |
| 🟢 | act completes Login Form | 1 step · 9 calls | 1m 22s |
| 🟢 | act completes Infinite Scroll List | 1 step · 3 calls | 1m 1s |
| 🟢 | act completes Modal Flow | 1 step · 4 calls | 17.2s |
| 🟢 | act completes Bottom Tabs | 1 step · 3 calls | 14.1s |
| 🟢 | act completes Text Input Variations | 1 step · 3 calls | 32.5s |
| 🟢 | act completes Broken Accessibility | 2 steps · 7 calls | 40.3s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 36.9s |
| act completes Gestures (1 failed attempt first) | 1 step · 10 calls | 48.5s | |
| 🟢 | act completes Async States | 1 step · 8 calls | 36.9s |
| 🟢 | act completes Debounced Search | 1 step · 4 calls | 22.2s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 19 calls | 2m 31s |
| 🟢 | act completes Flattened Login | 2 steps · 15 calls | 1m 33s |
| 🟢 | act completes Sticky Chrome Target | 1 step · 8 calls | 42.2s |
| 🟢 | act completes WebView Accessibility | 1 step · 6 calls | 42.7s |
| 🟢 | act completes Permission Prompt | 1 step · 3 calls | 23.2s |
| 🟢 | act completes Photo Picker | 1 step · 4 calls | 28.7s |
| 🟢 | act completes Product Catalog | 1 step · 5 calls | 20.1s |
| 🟢 | act completes Error Recovery | 1 step · 6 calls | 20.0s |
| 🟢 | act completes Choice Controls | 1 step · 8 calls | 42.8s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 8 calls | 45.4s |
| 🟢 | act completes Apple Pay | 1 step · 4 calls | 25.0s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
e2e 0.17.0 · 16m 28s · ios-simulator · run artifacts
Contributor
🟢 e2e android: 54 passed, 1 skippedAll 55 tests in 7 files
e2e 0.17.0 · 8m 4s · android-emulator · run artifacts |
Contributor
🟢 e2e android agent: 20 passed24 agent steps · 117 model calls · 674.6k tokens (84% cached) · $0.0419 All 20 tests in 1 file
e2e 0.17.0 · 9m 29s · android-emulator · run artifacts |
Contributor
🟢 e2e ios: 52 passed, 1 skippedAll 53 tests in 7 files
e2e 0.17.0 · 9m 18s · ios-simulator · run artifacts |
okwasniewski
added a commit
to pvedula7/e2e
that referenced
this pull request
Oct 8, 2026
okwasniewski
added a commit
to pvedula7/e2e
that referenced
this pull request
Oct 8, 2026
okwasniewski
added a commit
to Ashfaqbs/e2e
that referenced
this pull request
Oct 8, 2026
okwasniewski
added a commit
to pvedula7/e2e
that referenced
this pull request
Oct 8, 2026
This was referenced Oct 8, 2026
SudoScientist161
pushed a commit
to SudoScientist161/e2e
that referenced
this pull request
Oct 9, 2026
…ted models at config load (tester-army#916)
Ethereal49
pushed a commit
to Ethereal49/e2e
that referenced
this pull request
Oct 12, 2026
…er-army#910) * fix(decision): forward an element's selected state to the model checked and expanded already reach the decision model through the element table and target criteria; selected did not, even though the engine already reports it in SemanticNode.states. On a tab bar every tab therefore looked identical, so the completion check for a tab switch landed near 50/50: a step that worked could still be blocked, and a step could be marked done before anything was tapped. Forward selected the same way checked already is, in actionSpace's element table and in targetCriterion. * test(decision): use scriptedDecision after tester-army#916 --------- Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for
@e2e-dev/decisionbefore its first publish in #825, from the pre-release stress test.experimental_decidenow.experimental_evaluateis deprecated in ai 7.0.128, andtypeSafeAi.decisionModel()(@ai-sdk/typesafe-ai 3.0.13) failed typecheck with TS2741doEvaluatemissing.modelis typedExclude<Experimental_DecisionModel, string>fromai, so decision models and the deprecated evaluation models both typecheck and run.aigoes from^7.0.0to^7.0.128, the first release that exportsexperimental_decide(checked against the 7.0.127 and 7.0.128 tarballs). On main the floor was wrong anyway:experimental_evaluatefirst shipped in 7.0.103. pnpm only warns on an unmet peer, so the package readsexperimental_decideoff theainamespace, and an olderaigetsINVALID_CONFIG: decisionExecutor() needs ai 7.0.128 or laterinstead of a module link error.@ai-sdk/provider@4.0.21pin, which installed next to the 4.0.22 thataibrings. Types come from theaipeer.decisionExecutor()checks the model at config load and throwsConfigurationError('INVALID_CONFIG')naming the fix, the wayweb()andeasSimulators()do. Main threw a rawTypeError(packages/decision/src/executor.ts:29).MODEL_UNAVAILABLEnames the env var the provider reads (Set TYPESAFE_AI_API_KEY to the decision model API key.). The name comes out of the SDK'sLoadAPIKeyErrortext and nothing else in that text is kept.decisionModel().errors.mdxlistsdecisionExecutor()underINVALID_CONFIG. The existing unreleased changeset is amended, not added to.Verified
Ran it locally: yes. Packed
e2e,@e2e-dev/web,@e2e-dev/decisioninto a scratch project (/tmp/rc-decision),ai@7.0.128(floor and npm latest are the same version today), testbed/todosapp, mock choice model. No real model key on this box.model: openrouter('inception/mercury-2.5')TypeError: Cannot read properties of undefined (reading 'includes')INVALID_CONFIG: decisionExecutor({ model }) got the language model openrouter/inception/mercury-2.5; pass an AI SDK decision model, e.g. decisionExecutor({ model: typeSafeAi.decisionModel('jev-latest') })model: 'openai/gpt-5'TypeError: ... (reading 'includes')INVALID_CONFIG: decisionExecutor({ model }) got the string "openai/gpt-5"; ...TypeError: ... (reading 'supportedQuestionTypes')INVALID_CONFIG: decisionExecutor({ model }) got undefined; ...tscondecisionExecutor({ model: typeSafeAi.decisionModel('jev-latest') })TS2741: Property 'doEvaluate' is missingevaluationModel()tootypeSafeAi.decisionModel()run with no keyMODEL_UNAVAILABLE: Set the evaluation model API key.(from source, not run)MODEL_UNAVAILABLE: Set TYPESAFE_AI_API_KEY to the decision model API key.ai@7.0.127^7.0.0accepts 7.0.100-7.0.102, whose import fails with a link error (stress test, ai@7.0.100)npm installfails with ERESOLVE onpeer ai@"^7.0.128"; forced with--legacy-peer-deps:INVALID_CONFIG: decisionExecutor() needs ai 7.0.128 or later; update the ai packageReal CLI, both model kinds, checks a todo and asserts on
/todos:Unsupported model, as a user sees it (exit 2):
New tests: unit tests for each rejected shape and the key message, a test with
experimental_decidemocked away, and type tests intests/types/options.tsfordecisionModel(),evaluationModel(), and both literal shapes.