Repository navigation
Conversation
|
👋 Thanks for opening this PR! A maintainer will review it soon. In the meantime, please make sure you've read CONTRIBUTING.md and filled out the PR template, including screenshots for any visual changes. |
|
Sorry @Parag-098, your pull request is larger than the review limit of 150,000 diff characters |
MrHarshalBane
left a comment
There was a problem hiding this comment.
Pull Request Review: BYOK Multi-Provider & Health Check Improvements
Thanks for putting together this comprehensive expansion of the BYOK system. Supporting multi-provider credentials (OpenAI, Gemini, Anthropic, Groq, NVIDIA NIM, OpenRouter), granular usage attribution, and strict isolation for candidate practice mock sessions is a great improvement.
Below are findings from reviewing the implementation, including a few runtime bugs that should be addressed before merging.
1. response.json() crash on non-JSON HTTP errors (services/provider_health.py:67)
async with httpx.AsyncClient(timeout=timeout) as client:
response = await client.get(url, headers=headers)
status_code = response.status_code
data = response.json() if response.content else {}Issue: When upstream providers or proxies (e.g., Cloudflare error pages or plain-text error responses on 401/403/502/503) return non-JSON content, response.json() raises json.JSONDecodeError. Because this executes before evaluating if status_code < 300: and the HTTP status checks below it, the unhandled exception is caught by the blanket except Exception: block and classified as a generic connection_error ("Provider health check failed.") rather than reporting invalid_api_key or HTTP status.
Recommendation: Safely guard JSON decoding:
data = {}
if response.content and "application/json" in response.headers.get("content-type", ""):
try:
data = response.json()
except Exception:
data = {}2. Asyncpg DataError on ISO timestamp strings (repositories/postgres/llm_credential_repository.py:180-181)
async def save_platform_health(self, provider: str, result: dict) -> None:
pool = await self._pool.get()
async with pool.acquire() as conn:
await conn.execute("""INSERT INTO platform_llm_provider_health(provider,model,status,auth_status,last_tested_at,last_successful_test_at,error_code,message,updated_at)
VALUES($1,$2,$3,$4,$5,$6,$7,$8,now()) ...""",
provider, result.get("model"), result.get("status"), result.get("auth_status"),
result.get("last_tested_at"), result.get("last_successful_test_at"), ...)Issue: check_provider() returns ISO formatted strings for last_tested_at and last_successful_test_at. asyncpg strictly enforces Python datetime instances for PostgreSQL timestamptz columns ($5, $6). Passing strings directly will raise asyncpg.exceptions.DataError at runtime.
Recommendation: Parse into datetime before binding:
last_tested = datetime.fromisoformat(result["last_tested_at"]) if result.get("last_tested_at") else None
last_success = datetime.fromisoformat(result["last_successful_test_at"]) if result.get("last_successful_test_at") else None3. Missing failure callback when single key is configured without fallback (services/llm_credential_service.py:398)
chain = _tracked_platform(self._credentials, user_id) if user_settings.get("fallback_to_platform", True) else None
for record in reversed(records):
...
chain = FallbackLLMProvider(primary, chain, _on_primary_failure) if chain else primaryIssue: When fallback_to_platform is false and there is only one key (or on the last record when chain starts as None), chain is assigned directly to primary without wrapping in FallbackLLMProvider. Consequently, _on_primary_failure is never registered. If that credential fails, its status in the database is never updated to FAILED.
4. AnthropicProvider.generate_structured array truncation & placement (providers/llm/__init__.py:58-116)
- Array Schemas: The regex/slice fallback:
assumes structured output is always an object
start = cleaned.find("{") end = cleaned.rfind("}") return json.loads(cleaned[start:end + 1])
{...}. If the target schema is a top-level array[...](e.g. lists of questions or rubric items), searching for{and}will slice invalid substrings or fail. - Architectural Organization:
AnthropicProvideris currently defined directly insideproviders/llm/__init__.pywith local imports inside methods. It should ideally live in its own file (providers/llm/anthropic.py), matchinggemini.py,openai.py, andgroq.py.
5. is vs == for Enum comparisons (api/dependencies.py:17, api/routes/jobs.py:30,44, api/routes/llm_credentials.py:89)
Using identity comparison is for enum values (principal.principal_type is PrincipalType.ADMIN) is fragile across deserialization boundaries. Use equality == or !=.
6. Frontend UI card re-render overwrites status message (pages/profile.html:281, 296)
In pages/profile.html, testing a key sets statusEl.textContent and immediately calls await renderApiKeysCard(profile). Because renderApiKeysCard replaces container.innerHTML, the status element is instantly destroyed before the user can view the feedback. Updating the DOM row in place or deferring the re-render resolves the flicker.
7. Migration & Schema Redundancies (schema.sql & migrations/)
- In
repositories/postgres/schema.sql, lines 53–57 and 81–82 repeatALTER TABLE ADD COLUMN IF NOT EXISTSfor columns already declared in theCREATE TABLEdefinitions directly above them. 20261102_02_provider_health.sqlrepeats column additions already present in20261102_byok_multi_credentials.sql. Cleaning these up will prevent migration noise.
8. PR Scope & Description
The PR description is currently empty despite including ~2,900 lines of diff that also deprecate /live-sessions/demo and introduce session upgrade endpoints. Adding a brief summary in the PR description will provide helpful context for maintainers and future audits.
|
@MrHarshalBane Thanks for the review, I will check it and do necessary changes. |
No description provided.