perf: optimize hot-path auth and proxy pool lookups - #80
Closed
mahdiwafy wants to merge 52 commits into
Closed
Conversation
…apacity warning note
…+ usageMetadata) in parseSSEToOpenAIResponse
The parseSSEToOpenAIResponse function only understood OpenAI SSE format
(choices[0].delta.content, root-level usage). Antigravity/Gemini SSE
chunks use a different schema:
data: {"response":{"candidates":[{...}],"usageMetadata":{...}}}
- Text content from response.candidates[0].content.parts[].text
- Reasoning from .parts[].thought
- Finish reason from candidate.finishReason
- Usage from response.usageMetadata (promptTokenCount, candidatesTokenCount,
thoughtsTokenCount, totalTokenCount)
Also moves the OpenAI usage extraction before the AG block so the
existing path is preserved for OpenAI-formatted chunks.
…age tracking Tool-call-only turns (agentic coding clients like Claude Code/Kiro sending 100+ tools) stream their real output entirely through delta.tool_calls (and delta.partial_json for Claude tool_use), with delta.content and delta.reasoning_content empty. totalContentLength in createSSEStream() only summed content/reasoning length, so these turns left it stuck at 0. That starved both usage fallbacks: - estimateUsage() (provider sent no/invalid usage at all) - the new hasZeroCompletionWithContent()/fixZeroCompletionUsage() path (provider sent completion_tokens: 0 despite real output) Net effect: usage logs showed "out=0" for most tool-heavy turns even though the client received a large tool_calls payload — matching the live claude-sonnet-5 report on VansRouter (a5a5b570 account/Shiteru). Changes: - stream.js: sum delta.tool_calls[].function.name/arguments length (both PASSTHROUGH and TRANSLATE modes) and delta.partial_json (Claude tool_use) into totalContentLength. - usageTracking.js: add hasZeroCompletionWithContent() and fixZeroCompletionUsage() to patch a provider-reported zero completion count without discarding its (usually accurate) prompt_tokens. - Wire the new fallback into all 4 usage-decision sites in stream.js. Verified: reproduced the exact bug shape in a new integration test (in=166912, completion_tokens: 0 finish chunk, pure tool_calls output) — now estimates a non-zero completion count instead of logging 0. Deployed the fix live to the running container and confirmed in production logs: claude-sonnet-5 tool-heavy turns now log out=44/56/322/etc instead of a long run of out=0. Separately observed (NOT fixed by this change, needs its own investigation): a distinct pattern of genuinely empty completions (response.content === "[Empty streaming response]", no tool_calls either) from the Shiteru-backed claude-sonnet-5 route once prompt size exceeds ~130k tokens — confirmed via requestDetails table, not a usage- counting artifact.
…ojectId persistence, DuckDuckGo browser profile - ModelRow: empty draft on Save now deletes the alias instead of silently no-op - allowedModels: isModelAllowed enforces global list even without apiKeyInfo; appendUserAliasEntries surfaces bare alias names; cache invalidated on alias/disabled mutations - v1/models: alias entries pass through with owned_by=alias - chat.js + chatCore + stream/nonStreaming: echo client-facing model id (alias) back in /v1/chat/completions response.model and SSE chunks - projectId: cache TTL 30d bound to Google account not OAuth token; skip re-fetch when DB already has projectId (seedProjectIdCache) - settingsRepo + systemInject + chatCore: injectDefaultSystemPrompt only when request has no system message yet - DuckDuckGo: undici Agent (HTTP/2) + stable Chrome 131 header set (Sec-CH-UA client hints, full Accept)、 replacing the bot UA - registry/index.js: register duckduckgo provider - duckduckgo.png icon
…h label) validate/route.js had two `case "agentrouter":` labels in the same switch. JS only runs the first match, so agentrouter connections were silently validated by the generic glm/kimi/minimax block (bare x-api-key + anthropic-version, no headers) instead of validateAgentRouterConnection()'s Claude-CLI spoof headers. AgentRouter gates on client fingerprint and returns 401 'unauthorized client detected' for any request missing the spoof headers/UA — which the dashboard's Add API Key flow then reported as 'Invalid API key' even for a valid, unrestricted key. Removed agentrouter from the shared fallthrough case so it reaches the dedicated block. Verified live against the real AgentRouter API: the old (buggy) header set gets a genuine 401 from the same key that gets a non-401/403 response through validateAgentRouterConnection(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ve claude-opus-4-7
…atalog NVIDIA NIM chat/completions returned errors for several registry models: - z-ai/glm-5.1 → HTTP 410 Gone (EOL 2026-07-02). This is the primary failure users hit — the model is globally retired. - moonshotai/kimi-k2.6 (+ none/low/medium/high thinking variants) → HTTP 404 "not found for account": catalog-listed but not served on standard NIM developer keys. suggestedModels also carried dead ids: - z-ai/glm-5.1 (410 EOL), qwen/qwen3.5-397b-a17b (404 account), qwen/qwen3.5-122b-a10b and mistralai/mistral-large-3-675b-instruct-2512 (absent from /v1/models entirely). Replaced with models verified against integrate.api.nvidia.com (present in /v1/models and returning 200 on a real completion): GLM 5.2, DeepSeek V4 Pro/Flash, MiniMax M2.7/M3, GPT-OSS 120B/20B, Llama 4 Maverick, Step 3.7 Flash, Nemotron 3 Ultra/Super 120B, Nemotron Super 49B v1.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…atalog NVIDIA NIM chat/completions returned errors for several registry models: - z-ai/glm-5.1 → HTTP 410 Gone (EOL 2026-07-02). This is the primary failure users hit — the model is globally retired. - moonshotai/kimi-k2.6 (+ none/low/medium/high thinking variants) → HTTP 404 "not found for account": catalog-listed but not served on standard NIM developer keys. suggestedModels also carried dead ids: - z-ai/glm-5.1 (410 EOL), qwen/qwen3.5-397b-a17b (404 account), qwen/qwen3.5-122b-a10b and mistralai/mistral-large-3-675b-instruct-2512 (absent from /v1/models entirely). Replaced with models verified against integrate.api.nvidia.com (present in /v1/models and returning 200 on a real completion): GLM 5.2, DeepSeek V4 Pro/Flash, MiniMax M2.7/M3, GPT-OSS 120B/20B, Llama 4 Maverick, Step 3.7 Flash, Nemotron 3 Ultra/Super 120B, Nemotron Super 49B v1.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wasm32 fallback was in the main dependencies block with a pinned 4.3.0 version, causing npm install to fail with EBADPLATFORM on x64 hosts (and any non-wasm32 CPU). Move it to optionalDependencies mirroring the better-sqlite3 pattern and bump the spec to ^4.3.3 to match the resolved @tailwindcss/oxide version. Drop the non-standard comment_better_sqlite3 metadata key while here.
Hardcoded dev origins forced every contributor into the same localhost setup. Read VANS_ALLOWED_DEV_ORIGINS (comma-separated) with the prior 127.0.0.1,localhost as the default. Prod is unaffected — Next 16 ignores allowedDevOrigins outside dev mode.
Race: resolvePrivilegedUserId().then(...) was scheduled without await in
transformRequest, so the first request always went out without the
x-gemini-api-privileged-user-id header (the .then microtask could not
drain before buildHeaders ran synchronously). Fix:
- Prewarm the cache at module load so it is ready before any request.
- Make transformRequest await resolvePrivilegedUserId() so the header
is present on the very first request. Await is safe because
BaseExecutor.execute now wraps transformRequest in
await Promise.resolve(...), keeping the sync-return contract for
every other executor.
Mutation: stripOpenAILeakFields/applyIncludeReasoningToGemini mutated the
input body in place, leaving loggers, error DB saves, and the 401/403
retry path reading a stripped body instead of the client's original.
Fix: structuredClone(body) at the start of transformRequest so the
original translatedBody stays pristine for downstream consumers.
…branch webSearchInject.js (125 lines) was never imported by chatCore, chat, or any executor — the live web-search path lives inline in chat.js. Drop the orphan module. Simplify the redundant provider.id === 'duckduckgo' check in search/index.js (registry already sets executor: 'duckduckgo'). Remove the unused bodyHasReasoning variable in chat.js that nothing read.
The /^[a-zA-Z0-9_.\-]+$/ regex and its hint strings were copied across two API routes and the dashboard page. Extract COMBO_NAME_REGEX, COMBO_NAME_HINT, COMBO_ALIAS_HINT into src/shared/constants/comboValidation.js. Behavior unchanged: same validation, same error messages, same status codes.
Catch-block console.log calls in API routes wrote errors to stdout
instead of stderr, mixing diagnostics with normal output. Convert them
to console.error across providers/[id]/models, v1beta/models
([GEMINI_NATIVE] telemetry), keys, models/alias, models/disabled,
proxy-pools/[id], and cli-tools/claude-settings. Remove the stray
console.log('Deno deployUrl:', deployUrl) in proxy-pools/deno-deploy
that leaked every freshly-minted relay URL to stdout.
When PUT /api/keys/[id] received an allowedModels/allowedProviders/ allowedCombos/allowedKinds value that was neither null nor an array (string, object, number), it silently coerced to null — meaning 'all allowed' — masking caller bugs as over-permissive defaults. Default unexpected types to [] (deny-all) for all four ACL fields, matching the documented contract: null = all, [] = none, [...] = specific.
The system-prompt save handler swallowed all failures silently (empty
catch), leaving users with no indication the save failed. Reuse the
file's existing {type, message} status pattern to surface errors next
to the Save button. Remove the unused isDark destructure from
useTheme() (never referenced in the file).
The chromium executablePath was pinned to /home/vanszs/.cache/ ms-playwright/chromium-1228/chrome-linux64/chrome — one contributor's absolute home path that fails on every other machine. Read VANS_ZCODE_CHROMIUM_PATH from the environment; pass undefined to let playwright-core use its own channel discovery when the env var is unset.
The dev prewarm script fell back to '123456' when VANS_PREWARM_PASSWORD was unset, silently authenticating with the default dashboard password in any environment that forgot to set the env var. Fail closed: if the env var is missing, print a clear error and exit non-zero so the operator sets it explicitly.
The per-process _binCache/_pyCache/_extrasCache never exposed an invalidation path, so installing headroom-ai while the server was running left the dashboard reporting 'not installed' until process restart. Export invalidateHeadroomCaches() so the install endpoint can reset the probes after a successful install.
Extend the shared COMBO_NAME_REGEX to ComboFormModal and the media-providers combo edit page, the two remaining copies of the /^[a-zA-Z0-9_.\-]+$/ regex that were protected by the first DRY pass. Behavior unchanged.
The /api/models response is always { models: [...] } by convention, so
the Array.isArray(data) fallback at line 526 was dead code reachable
only if the endpoint changed shape. Remove it. Restore 'No providers'
capitalization to match the adjacent 'No models' badge.
The commit that converted login-page strings from inline Bahasa to inline English left Indonesian-cookie users seeing English, because no corresponding keys existed in public/i18n/literals/id.json and the runtime DOM translator only matches known English source strings. Wrap the placeholder, dispatch error fallbacks, button label, and OIDC fallback in translate() from @/i18n/runtime (attributes and state-driven strings are not reached by the MutationObserver-driven DOM walker). Add the missing English keys to id.json, using the pre-commit Bahasa strings as the Indonesian translations.
claude.ai/oauth/authorize rejects our redirect_uri (https://api.bevansatria.my.id/callback) with "Redirect URI ... is not supported by client" because it isn't on the whitelist for the public Claude Code client_id. Switch to the same redirect Anthropic's own Claude Code CLI uses (https://platform.claude.com/oauth/code/callback), which is whitelisted. Since that redirect lands on Anthropic's domain instead of our own /callback route, the popup can never postMessage/BroadcastChannel the code back to us — force manual-paste mode for the claude provider and accept Anthropic's bare `code#state` callback format (not a full URL). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Claude Code (cc) provider's model list stopped at Opus 4.8 and was missing the entire Claude 5 generation — even though the CLI defaults in cliTools.js already point at cc/claude-fable-5 and cc/claude-sonnet-5. Add the three generally-available Claude 5 models (per Anthropic's model docs), keeping 4.x as legacy entries: - claude-fable-5 ($10/$50, 1M ctx, adaptive thinking always-on) - claude-opus-5 ($5/$25, 1M ctx, adaptive thinking) - claude-sonnet-5 ($3/$15, 1M ctx, adaptive thinking) Also add exact capabilities entries for claude-opus-5 / claude-fable-5 (so the generic *claude* pattern doesn't downgrade them to claude-budget thinking) and canonical pricing for claude-opus-5 / claude-sonnet-5. claude-mythos-5 is intentionally omitted — it is invitation-only (Project Glasswing) and not generally accessible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match the official Claude Code OAuth client scope set by requesting user:sessions:claude_code and user:mcp_servers in addition to the existing profile/inference scopes. Tokens minted without these scopes can connect for basic OAuth but fail Claude Code session/usage endpoints or org policy checks. Also send the same Claude CLI fingerprint headers to the OAuth usage endpoint that we already use for Claude Code inference. When Anthropic rejects OAuth for an organization, surface an explicit org-policy message instead of falling back to the misleading legacy admin-permissions text. Note that oauth_not_allowed_for_organization is still an Anthropic org policy decision; users must reconnect after this change to mint a fresh scoped token, and accounts in orgs that disable OAuth still need an org admin or a different Claude account. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Gemini CLI fixes
…rnal PR Vanszs#61's Gemini CLI fix converts parameters -> parametersJsonSchema for Cloud Code Assist. But Antigravity's v1internal API uses the standard 'parameters' field. The ...fn spread in transformRequest leaked parametersJsonSchema through while also setting parameters, causing Google API to reject with: parameters_json_schema must not be set when parameters is set. Delete parametersJsonSchema after building each function declaration in the antigravity executor tool pipeline.
…rnal PR Vanszs#61's Gemini CLI fix converts parameters -> parametersJsonSchema for Cloud Code Assist. But Antigravity's v1internal API uses the standard 'parameters' field. The ...fn spread in transformRequest leaked parametersJsonSchema through while also setting parameters, causing Google API to reject with: parameters_json_schema must not be set when parameters is set. Delete parametersJsonSchema after building each function declaration in the antigravity executor tool pipeline.
…PI key limits - Bundle tailscale/tailscaled binaries in Docker image and install to /usr/local/bin so they survive the /app/data volume mount. Add TAILSCALE_USE_HOST_SOCKET toggle to reuse the host's Tailscale daemon via a mounted socket. - Trust router.mahdiwafy.my.id for public /v1/* API access via PUBLIC_API_HOSTS and add detailed dashboardGuard logging for remote API-key decisions. - Add optional per-key limits: expiresAt, maxTokens, maxTokensDaily, rpm, rph, rpd, tokens5h, tokensWeekly, tokensMonthly. Enforce at request time and expose in dashboard key editor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Extends the proxy rotation strategy feature to also apply in the non-free (connection-based) provider path. After selecting a connection, if the provider has a rotateStrategy configured (round-robin or random), the proxy pool is dynamically selected from all available pools instead of using the connection's static proxy pool binding.
- New passkeys table & migration v5 (credential ID, publicKey, counter, etc.) - WebAuthn ceremony API routes: register start/finish, login start/finish, manage - IP detection via isLocalRequest: remoteAuthMode controls login options remotely - Login page: passkey button when passkeys are available - Masuk page: same passkey support in Indonesian locale - Profile page: full passkey management (register, list, delete, remote mode selector) - Uses @simplewebauthn/server + @simplewebauthn/browser v13
… rebuild on CI failure
Author
|
Closing — this PR accidentally included all fork changes. Will reopen with a clean branch containing only the perf optimization commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimize middleware and repository hot paths to reduce per-request latency:
Changes
src/dashboardGuard.js - Reduce auth logging noise in hot path
DEBUG_AUTHenvPUBLIC_API_HOSTS) logic extracted and clarifiedsrc/lib/db/repos/apiKeysRepo.js - Add 2s in-memory validation cache
validateApiKey()now caches hits for 2 secondsexpiresAtif soonersrc/lib/db/repos/proxyPoolsRepo.js - Add 2s in-memory proxy pool cache
getProxyPools()andgetProxyPoolById()cached for 2 secondssrc/lib/db/repos/connectionsRepo.js - Replace uuid dependency
crypto.randomUUID()instead ofuuidpackageVerification
npm run lint:undef— cleannpm run build— exit 0vansrouter:optimized-auth-proxy-cache)vansrouteron port 20128 — healthy/v1/modelswith API key: avg 0.0075s (cached)/v1/chat/completionsvia OPENCODE: HTTP 200 in ~3.2s