Stabilization and better retry

[yhurynovich]

Main Proxy Server & System Fixes Documentation

Overview of Changes & Fixes

server.js - Main Proxy Server

Area	Original	Fixed
Startup	Fire-and-forget initializeKeys().catch()	Blocking await initializeKeys() - server won't accept requests until keys loaded
Multiple Keys	Only reads single OPENROUTER_API_KEYS	Splits comma-separated OPENROUTER_API_KEYS into multiple keys
Axios Config	No timeout, no trust_env	120s timeout + trust_env: false (ignores system proxy)
Non-streaming Response	Returns response.data directly (even if error in body)	Validates response body for error.message with NVIDIA patterns → throws to trigger retry
Streaming Response	Pipes chunks directly to client	Buffers & inspects chunks for error.message with NVIDIA patterns → throws to trigger retry
Error Handling	Streaming: ends stream immediately, no retry	Both streaming & non-streaming: detects NVIDIA rate limit → waits 1 second → retries up to 3 times with rotated key
NVIDIA Detection	None	Case-insensitive patterns: Upstream error from Nvidia + ResourceExhausted / rate limit / limit reached

---

services/KeyManager.js - Key Rotation Logic

Area	Original	Fixed
NVIDIA Detection	Only ResourceExhausted	Multiple patterns: ResourceExhausted OR rate limit OR limit reached
Rate Limit Handling	Only HTTP 429	HTTP 429 OR NVIDIA error in response body
Key Rotation	Sets cooldown on rate limit	Sets cooldown + immediately rotates to next key
Failure Tracking	Increments on ALL errors	Only increments on non-rate-limit errors (prevents deactivating keys for transient NVIDIA limits)
Logging	Basic	Logs isNvidia: true for debugging

---

services/utils/sanitize.js - Log Redaction

Area	Original	Fixed
Fields Redacted	Only Authorization header + apiKey in body	All common API key fields: Authorization, x-api-key, api_key, key in body

---

models/ApiKey.js - Identified Issues

Issue	Status
No file locking for concurrent access	Known limitation - JSON file corruption possible under high concurrency
Reads entire file on every query	Known limitation - inefficient for many keys

---

Key Behavioral Changes

• ORIGINAL:
  $$\text{Request} \longrightarrow \text{NVIDIA returns 200 with error in body} \longrightarrow \text{Client sees error}$$

• FIXED:
  $$\text{Request} \longrightarrow \text{Detect NVIDIA error} \longrightarrow \text{Rotate key} \longrightarrow \text{Wait 1s} \longrightarrow \text{Retry (}\times 3\text{)} \longrightarrow \text{Success}$$

Summary: The proxy now transparently handles NVIDIA rate limits — clients never see the "ResourceExhausted" error.

---

NVIDIA Error Formats Handled

The proxy inspects and detects errors matching these structures (e.g. from system logs):

In SSE Stream Chunk

{
  "error": {
    "message": "Upstream error from Nvidia: ResourceExhausted: Worker local total request limit reached (32/32)"
  }
}

In Non-Streaming Response

{
  "error": {
    "message": "Upstream error from Nvidia: ResourceExhausted: Worker local total request limit reached (32/32)"
  }
}

---

Environment Variables Supported

Multiple Keys (Comma-Separated)

OPENROUTER_API_KEYS="key1,key2,key3"

Optional Configuration

PORT=3000
HTTP_REFERER=http://localhost:3000
SITE_NAME=OpenRouterProxy