Skip to content

fix(providers): repair the AI/ML API integration — 6 stale models to 353 live, and 400s on every message - #1

Open
Lookoff-AIMLAPI wants to merge 5 commits into
release/v3.8.51from
fix/aimlapi-model-discovery
Open

fix(providers): repair the AI/ML API integration — 6 stale models to 353 live, and 400s on every message#1
Lookoff-AIMLAPI wants to merge 5 commits into
release/v3.8.51from
fix/aimlapi-model-discovery

Conversation

@Lookoff-AIMLAPI

Copy link
Copy Markdown
Member

Summary

The aimlapi provider is fully wired in this repo and has been showing users 6 models, 4 of
which no longer exist upstream
. It should be showing 353. And once discovery is fixed, most
of those models still 400 on every message, for a second and unrelated reason. This PR fixes both,
and tags the resulting traffic with partner attribution.

⚠️ base-red inherited: diegosouzapw#12518

Defect 1 — discovery returns zero models

Verified against the live catalog on 2026-09-03 (GET https://api.aimlapi.com/models — public, no
key; 936 rows, 353 of them chat):

# Where Defect
1a discovery/providerModelsConfig.ts Array.isArray(data) against a response that is now the OpenAI-style envelope {"object":"list","data":[…]}. Always false → every row discarded → discovery returned [].
1b same chat filter looked for type === "chat-completion"; the catalog publishes openai/chat-completions. The old spelling matches 0 of 936 rows.
1c same chat.length ? chat : all fall-through would, once 1b started missing, have published 583 video/image/TTS/batch ids into a chat model picker — which is what kept 1b invisible.

1a alone is fatal: parseResponse returns [], buildApiDiscoveryResponse takes its
discoveredModels.length > 0 false branch, and the route re-derives from the static seed. The user
sees exactly the 6 seed entries. passthroughModels: true means a hand-typed id still routes; it is
the browsable list that was dead.

parseResponse against the real live catalog: 0 models before, 353 after.

Both the response shape and the type vocabulary changed upstream without a deprecation path, so
the parser now accepts both spellings and unwraps either shape — a further rename degrades to
"some models missing" rather than "the provider has no models".

Defect 2 — 400 on every message, even with a full model list

Independent of the above, and it would have survived the discovery fix: AI/ML API validates
optional fields with a strict schema and answers 400 "Expected number, received null" when a
field arrives as a literal null, rather than reading null as "unset". OmniRoute relays the
caller's body verbatim, and the OpenAI SDKs serialise an unset optional as null — so an SDK
client that never sets temperature still sends temperature: null and gets a 400 on every turn.
Fixing discovery alone would have handed those users a 353-model picker attached to a provider that
still could not answer.

Field-by-field sweep against POST /v1/chat/completions, 2026-09-03:

behaviour on null fields
400, every model seed, tools, tool_choice, response_format, stream, stream_options, parallel_tool_calls, max_tokens, max_completion_tokens, reasoning_effort
400, model-dependent temperature, top_p — 400 on claude-sonnet-4.6 and deepseek-chat, 200 on gpt-5
200 (left alone) stop, presence_penalty, frequency_penalty, n, user, logprobs, logit_bias, top_logprobs, metadata

Two things make this expensive to diagnose from the outside: temperature/top_p are
model-dependent, so an integration smoke-tested against gpt-5 looks healthy; and the 400 body's
top-level message is generic — only details[].path / .reason name the offending field. The
list above was read off details[].path.

Fixed as a dropIfNull rule in the existing STRIP_RULES table (translator/paramSupport.ts),
whose stated purpose is "params a given provider/model rejects upstream". No new mechanism, and it
stays provider-scoped. Dropping is the correct reading — every field listed means "unset" when the
caller sends null. dropIfNull fires only on a literal null, which is the point: a plain drop
would also discard a deliberate temperature: 0, stream: false, parallel_tool_calls: false or
tools: [].

Defect 3 — the seed model ids

Every id was checked twice, because neither check alone is sufficient: present on the catalog's
chat surface as an id OR alias, and answering 200 to a real POST /v1/chat/completions.

Removed Why Replaced with
gpt-4o live alias, not dead — replaced only for consistency gpt-5
claude-3-5-sonnet-20241022 absent as id and alias claude-sonnet-4.6
gemini-1.5-pro absent as id and alias gemini-2.5-pro
meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo absent as id and alias glm-5
deepseek-chat live alias, not dead — kept deepseek-chat
mistral-large-latest absent as id and alias mistral-large

Three things this turned up that are worth knowing before anyone edits this list again:

  • Catalog membership does not imply routability. llama-3.3-70b-versatile is listed as a chat
    model, advertises tools and structured output, and still returns
    404 "The model does not exist or you do not have access to it" on inference. It was in an
    earlier draft of this seed purely on catalog evidence.
  • Seed ids must stay unprefixed. A vendor/model id in this registry is matched by
    parseModel() as an exact model id, which then reports provider = null. An earlier draft seeded
    anthropic/claude-sonnet-4-6 and that string stopped resolving to the anthropic provider
    app-wide — context window collapsed from 1M to the 128k default, and combo targets lost the
    provider prefix. combo-target-token-limit-8716 caught it. The previous seed's bare ids were
    right about this.
  • Anthropic ids need the dotted spelling. The catalog carries claude-sonnet-4-6 and
    claude-sonnet-4.6 as separate entries; the dashed one advertises only streaming in
    capabilities while the dotted one advertises tools, vision, reasoning and structured output.
    Same split for claude-opus-4.7 / 4.8 — 3 such pairs out of 7.

Why the tests did not catch any of this

Both existing discovery tests fed the parser a bare array of chat-completion rows — a shape
the endpoint has not returned since the schema change. They passed for the entire outage. They now
assert the real envelope and the real type value, plus new cases for the legacy spellings and for
the "never fall through to non-chat rows" rule.

Attribution

Four headers on the aimlapi registry entry, using the mechanism already present for
openrouter / orcarouter / cline / gitlawb:

HTTP-Referer:         https://github.com/diegosouzapw/OmniRoute
X-Title:              OmniRoute
X-AIMLAPI-Source:     agent/omniroute
X-AIMLAPI-Partner-ID: part_omniroute

BaseExecutor.buildHeadersPreamble() spreads config.headers into a freshly built map per request,
so they are scoped to this provider's dispatch (they cannot ride a request to another upstream) and
the shared registry constant is never mutated — both asserted, and the regenerated translate-path
golden shows the four headers under aimlapi and under no other provider.

HTTP-Referer / X-Title follow the OpenRouter convention and name the calling app, not the
gateway. The partner id is asserted against ^part_[A-Za-z0-9]{1,64}$, because a malformed id has
no runtime symptom — the gateway accepts the request and records the usage as untagged.

Display label is now aimlapi.com. The machine id (aimlapi) and alias (aiml) are unchanged;
those are what existing configs and stored connections key on.

The last commit is fork-only

chore(aimlapi): fork-only placement — do not send upstream pins aimlapi.com in the dashboard
grids. Drop that commit before offering anything upstream. It is isolated and carries its own
guard test so it comes out in one piece.

It ranks aimlapi 3rd, below the two sponsors the operator ranked explicitly on 2026-07-31, and
adds no supporter chip — rendering one would assert an "Open Source Friend" sponsorship that
does not exist. The provider catalog's key order is untouched: filterConfiguredProviderEntries()
sorts every grid by display name and PROVIDER_REFERENCE.md is generated alphabetically, so the
rank map is the only lever that reaches the rendered order.

Related Issues

Validation

  • Change type: provider
  • Focused tests from the golden path (see below)
  • npm run lint — clean on every changed file
  • npm run check:docs-all — exit 0
  • Reconciled with release/v3.8.51
  • Production-code changes include new/updated automated tests in this PR

Unit suite — same command, same host, before and after:

tests pass fail skipped
baseline (pristine release/v3.8.51) 36743 36709 8 23
with this branch 36765 36731 9 22

+22 tests, +22 passing — the new and updated files. The failing set is the baseline set plus
exactly one
, and every member of it is pre-existing and unrelated to providers: event-loop timing
(9147-catalog-eventloop-yield), Redis-dependent suites (quota-redis-store,
rate-limit-manager), an unreleased DB handle (combo-context-overflow-compression-probe), and
shell/binary-manager tests. The base branch is independently confirmed red:
diegosouzapw#12518.

The one extra is resolve-npm-entry → "live environment: the real node install can resolve
npm-cli.js". It is a host-layout probe of scripts/build/resolveNpmEntry.ts, a file this PR does
not touch: Homebrew keeps npm at /opt/homebrew/lib/node_modules/npm/bin/npm-cli.js while the
resolver derives /opt/homebrew/Cellar/node/26.7.0/lib/node_modules/… from process.execPath. It
fails deterministically on this machine (2/2 standalone runs) and will not reproduce on CI's runner
layout.

Vitest UI: ProviderIcon-icon-url, providerCardKimiPartnerAccent,
providerPageHeaderKimiPartnerLink — 3 files / 101 tests pass.

Production build: npm run build fails on this branch and on a pristine base for a reason
neither introduced nor touched here: docs/reference/REMOVED_PROVIDERS.md has no YAML frontmatter,
so the MDX pipeline throws title: Invalid input: expected string, received undefined. It is the
only file under docs/reference/ missing frontmatter, it arrived in diegosouzapw#12478 one commit before the
base tip, and it is the "Turbopack build failed" hard failure in diegosouzapw#12518. With that single
pre-existing defect temporarily patched out locally, npm run build exits 0 with this branch's
changes
. Not fixed here, per the base-red rule that such fixes belong in their own PR.

Live verification — real inference through the integration

Not a mock and not a raw curl: through getDefaultExecutor("aimlapi")DefaultExecutor.execute(),
the registry entry and header path this PR changes.

== discovery ==
GET https://api.aimlapi.com/models -> 200   raw rows: 936
parseResponse -> 353 models        (old parser on the same payload -> 0)

== headers actually dispatched ==
{"Content-Type":"application/json",
 "HTTP-Referer":"https://github.com/diegosouzapw/OmniRoute",
 "X-Title":"OmniRoute",
 "X-AIMLAPI-Source":"agent/omniroute",
 "X-AIMLAPI-Partner-ID":"part_omniroute",
 "Authorization":"<redacted>",
 "Accept":"application/json"}

== live inference ==
POST https://api.aimlapi.com/v1/chat/completions -> 200
{"model":"meta-llama/Llama-3.3-70B-Instruct-Turbo",
 "choices":[{"finish_reason":"stop","message":{"role":"assistant",
   "content":"OmniRoute aimlapi repair OK","tool_calls":[]}}],
 "usage":{"prompt_tokens":46,"completion_tokens":9,"total_tokens":55}}

== tool calling ==
POST /v1/chat/completions -> 200
{"choices":[{"finish_reason":"tool_calls","message":{"role":"assistant","content":null,
  "tool_calls":[{"id":"call_6mmztefjwlbnnwzldzrljfx8","type":"function",
    "function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"}}]}}]}

== null-param fix, same code path, payload shaped like an OpenAI SDK's ==
before: 12 optional fields sent as null                      -> 400 "Expected number, received null"
after:  body OmniRoute sent:
        {"model":"claude-sonnet-4.6","messages":[…],"stop":null,"presence_penalty":null,
         "n":null,"user":null,"max_tokens":32,"stream":false}
        -> 200, content "OmniRoute aimlapi repair OK"
        (the 12 rejected nulls dropped; the 4 the upstream accepts as null kept verbatim)

Tests Added Or Updated

  • tests/unit/aimlapi-catalog-repair.test.ts (new) — seed ids, known-dead-id blocklist (now
    including the catalog-listed-but-404 id and the dashed Anthropic spelling), no-vendor-prefix
    guard, endpoint shape
  • tests/unit/aimlapi-attribution-headers.test.ts (new) — the four headers, partner-id regex,
    provider scoping, no-mutation of the shared constant, display label
  • tests/unit/aimlapi-null-param-strip.test.ts (new) — all 12 rejected fields dropped, falsy-but-real
    values (temperature: 0, stream: false, tools: []) preserved, the 9 fields the upstream accepts
    as null left alone, the two lists asserted disjoint, no leakage to other providers
  • tests/unit/aimlapi-fork-placement.test.ts (new, fork-only commit) — dashboard pin
  • tests/unit/provider-models-discovery-split.test.ts (updated) — envelope, both type spellings,
    bare-array back-compat, no fall-through to non-chat rows
  • tests/unit/provider-models-route.test.ts (updated) — mock now returns the real envelope
  • tests/unit/executors-strip-unsupported-params.test.ts (updated) — the STRIP_RULES shape guard
    now accepts a dropIfNull-only rule

Coverage Notes

Four production files change (providerModelsConfig.ts, registry/aimlapi/index.ts,
apikey/gateways.ts, translator/paramSupport.ts; plus featuredProviders.ts in the fork-only
commit). Each is covered above. The two edited discovery tests are RED against the pre-change parser
(it returns [] for the real envelope) and the null-param tests are RED against the pre-change
rule table, so these are genuine regression guards rather than restatements.

Reviewer Notes

  • Drop the fork-only commit before any upstream PR. It is the only self-promotional change here.
  • Discovery accepts both chat type spellings on purpose, so a further rename degrades to "some
    models missing" rather than the failure mode being fixed.
  • The dropIfNull rule is deliberately provider-scoped. A global null-strip is arguably correct
    (the OpenAI schema treats null as unset) but that is a behaviour change for 355 providers and
    belongs in its own PR with its own evidence. Other OpenAI-compatible upstreams in this registry
    may well have the same intolerance; nothing here surveys them.
  • Not verified: nothing was exercised through a running dashboard; verification was at the
    discovery-parser and executor level plus live upstream calls. Streaming was not exercised live,
    only the non-streaming path.
  • Worth flagging platform-side: GET /v1/models returns 200 for any key, including a bogus one,
    so the common "validate the key by GETting /models" pattern silently accepts garbage here. This
    repo already dodged that trap for OpenRouter via testKeyModelsUrl
    (registry/openrouter/index.ts:12-16); AI/ML API exposes no equivalent authenticated endpoint to
    point at, so none is added.

… catalog schema

AI/ML API has been discovering zero models since its catalog changed shape
upstream, so every user of the provider saw the 6-entry static seed instead of
the live list — and four of those six ids no longer exist, so most of what was
offered 404'd on first use.

Three independent defects, each verified against the live catalog on 2026-09-03
(936 rows, 353 of them chat):

- The parser tested `Array.isArray(data)` against a response that is now the
  OpenAI-style envelope `{ "object": "list", "data": [...] }`. Every row was
  discarded before any filter ran, so discovery returned [] and the route took
  its local_catalog branch. Sibling entries (thebai, openrouter) already unwrap
  the envelope themselves; this one never did.
- The chat filter looked for `type === "chat-completion"`, a spelling the
  catalog no longer publishes. The current value is `openai/chat-completions`
  and the old one matches 0 of 936 rows. Both are accepted now, so a further
  rename degrades to "some models missing" rather than "no models at all".
- The `chat.length ? chat : all` fallback would, once the type filter stopped
  matching, have published 583 video/image/TTS/batch ids into a chat model
  picker. Dropping it is what makes defect 2 observable instead of silent.

The static seed replaces the four dead ids (claude-3-5-sonnet-20241022,
gemini-1.5-pro, meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo,
mistral-large-latest — absent from the catalog as both id AND alias) with six
current ones. Each replacement was checked twice, because neither check alone is
sufficient: it must appear on the catalog's chat surface as an id or an alias,
AND answer 200 to a real chat completion. `llama-3.3-70b-versatile` is why —
it is listed as a chat model, advertises tools and structured output, and still
404s "model does not exist" on inference.

The seed keeps the previous list's UNPREFIXED spelling. A `vendor/model` id in
this registry is matched by parseModel() as an exact model id, which makes it
report provider = null: seeding "anthropic/claude-sonnet-4-6" here stops that
string resolving to the anthropic provider anywhere in the app, collapsing its
context window from 1M to the 128k default and dropping the provider prefix that
diegosouzapw#8716 exists to preserve. That regression is not hypothetical — an earlier draft
of this change caused it, and combo-target-token-limit-8716 caught it.

Anthropic ids use the dotted spelling. The catalog carries `claude-sonnet-4-6`
and `claude-sonnet-4.6` as separate entries and the dashed one advertises only
`streaming` in its capabilities, while the dotted one advertises tools, vision,
reasoning and structured output. The same split exists for claude-opus-4.7/4.8.

Both existing tests passed throughout the outage because their mocks fed the
parser a bare array of `chat-completion` rows — a shape the endpoint has not
returned since the change. They now assert the real envelope, which is what
turns this from a green suite over a broken provider into a regression guard.
…se its own brand name

OmniRoute's AI/ML API traffic currently reaches the gateway untagged, so none of
it is attributable to this project. Four headers fix that, using the mechanism
the registry already has rather than any new machinery: openrouter, orcarouter,
cline and gitlawb all declare a `headers` block on their registry entry, and
BaseExecutor.buildHeadersPreamble() spreads it into a freshly built map per
request. That gives the two properties this needs for free — the headers are
scoped to this provider's own dispatch, so they can never ride a request to
another upstream, and the shared registry constant is never mutated. The
regenerated translate-path golden shows exactly that: the four headers appear
under `aimlapi` and under no other provider.

HTTP-Referer and X-Title follow the OpenRouter convention and name the CALLING
application, so they point at OmniRoute's own repository, not at the gateway
being called.

The partner id is asserted against `^part_[A-Za-z0-9]{1,64}$` in a test because
a malformed one has no runtime symptom: the gateway accepts the request either
way and simply records the usage as untagged, so a typo would cost attribution
silently and forever.

The dashboard label becomes `aimlapi.com`, the name the provider ships under.
The machine identifier (`aimlapi`) and alias (`aiml`) are untouched — those are
what existing user configs and stored connections key on, and renaming them
would break them.
…to AI/ML API

A second live defect, independent of the discovery bug and one that would have
survived fixing it: AI/ML API validates optional fields with a strict schema and
answers 400 "Expected number, received null" when a field arrives as a literal
`null`, instead of reading null as "unset". OmniRoute relays the caller's body
verbatim, and the OpenAI SDKs serialise an unset optional as `null` — so an SDK
client that never sets temperature still puts `temperature: null` on the wire and
gets a 400 on every single message. Repairing discovery alone would have handed
those users a 353-model picker attached to a provider that still could not answer.

Field-by-field sweep against POST /v1/chat/completions on 2026-09-03, read off
`details[].path` in the 400 bodies (the top-level `message` is generic and names
no field, which is most of why this is hard to diagnose from a log):

  400 on null, every model  seed, tools, tool_choice, response_format, stream,
                            stream_options, parallel_tool_calls, max_tokens,
                            max_completion_tokens, reasoning_effort
  400, model-dependent      temperature, top_p — 400 on claude-sonnet-4.6 and
                            deepseek-chat, 200 on gpt-5, so an integration
                            smoke-tested only against gpt-5 looks healthy
  200 on null (untouched)   stop, presence_penalty, frequency_penalty, n, user,
                            logprobs, logit_bias, top_logprobs, metadata

Expressed as a `dropIfNull` rule in the existing STRIP_RULES table, whose stated
purpose is "params a given provider/model rejects upstream" — no new mechanism,
and it stays provider-scoped. Dropping is the correct reading: every field listed
means "unset" when the caller sends null.

`dropIfNull` fires only on a literal null, which is the whole point: a plain
`drop` would also discard a deliberate `temperature: 0`, `stream: false`,
`parallel_tool_calls: false` or `tools: []`. Those are covered by a test.
Pins aimlapi.com in the dashboard provider grids for this fork only. It is
isolated in a single commit, with its guard test, so it can be dropped whole
before anything is offered upstream: a placement request without a partnership
behind it is the wrong thing to put in front of a maintainer, and it has no
business travelling with the functional repair.

The rank map is the only lever that reaches the rendered order.
filterConfiguredProviderEntries() sorts every grid by display name and
docs/reference/PROVIDER_REFERENCE.md is generated alphabetically, so the
provider catalog's key order never surfaces to a user and is left untouched.

Ranked 3, below Kimi (1) and Cheaper Inference (2). Those two encode an explicit
operator decision dated 2026-07-31 and are asserted in
featured-providers-rank.test.ts; reordering them here would mean editing someone
else's stated commitment to move ourselves above it. Rank 3 still pins us above
every other aggregator in the grid.

No supporter chip is added. ProviderCard renders those from per-sponsor
predicates tied to the "Open Source Friend" programme, and rendering one would
assert a sponsorship that does not exist.
The placeholder part_omniroute was a readable stand-in chosen before the
partner was registered. Registration mints the id server-side, so the
real value is part_T2iNtMuQ3JBmEPwyOKCLOxaP. A wrong or unknown partner id is accepted with a
200 and silently not attributed, so this would not have surfaced at runtime.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant