Skip to content

Implement BYOK improvement - #80

Open
Parag-098 wants to merge 3 commits into
DevSidd2006:mainfrom
Parag-098:feature/byok-improvements
Open

Parag-098 wants to merge 3 commits into
DevSidd2006:mainfrom
Parag-098:feature/byok-improvements

Conversation

@Parag-098

Copy link
Copy Markdown
Contributor

No description provided.

Copilot AI balanced review requested due to automatic review settings November 2, 2026 13:14
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

👋 Thanks for opening this PR! A maintainer will review it soon.

In the meantime, please make sure you've read CONTRIBUTING.md and filled out the PR template, including screenshots for any visual changes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sourcery-ai

sourcery-ai Bot commented Nov 2, 2026

Copy link
Copy Markdown

Sorry @Parag-098, your pull request is larger than the review limit of 150,000 diff characters

@MrHarshalBane MrHarshalBane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Review: BYOK Multi-Provider & Health Check Improvements

Thanks for putting together this comprehensive expansion of the BYOK system. Supporting multi-provider credentials (OpenAI, Gemini, Anthropic, Groq, NVIDIA NIM, OpenRouter), granular usage attribution, and strict isolation for candidate practice mock sessions is a great improvement.

Below are findings from reviewing the implementation, including a few runtime bugs that should be addressed before merging.


1. response.json() crash on non-JSON HTTP errors (services/provider_health.py:67)

async with httpx.AsyncClient(timeout=timeout) as client:
    response = await client.get(url, headers=headers)
status_code = response.status_code
data = response.json() if response.content else {}

Issue: When upstream providers or proxies (e.g., Cloudflare error pages or plain-text error responses on 401/403/502/503) return non-JSON content, response.json() raises json.JSONDecodeError. Because this executes before evaluating if status_code < 300: and the HTTP status checks below it, the unhandled exception is caught by the blanket except Exception: block and classified as a generic connection_error ("Provider health check failed.") rather than reporting invalid_api_key or HTTP status.

Recommendation: Safely guard JSON decoding:

data = {}
if response.content and "application/json" in response.headers.get("content-type", ""):
    try:
        data = response.json()
    except Exception:
        data = {}

2. Asyncpg DataError on ISO timestamp strings (repositories/postgres/llm_credential_repository.py:180-181)

async def save_platform_health(self, provider: str, result: dict) -> None:
    pool = await self._pool.get()
    async with pool.acquire() as conn:
        await conn.execute("""INSERT INTO platform_llm_provider_health(provider,model,status,auth_status,last_tested_at,last_successful_test_at,error_code,message,updated_at)
            VALUES($1,$2,$3,$4,$5,$6,$7,$8,now()) ...""",
            provider, result.get("model"), result.get("status"), result.get("auth_status"),
            result.get("last_tested_at"), result.get("last_successful_test_at"), ...)

Issue: check_provider() returns ISO formatted strings for last_tested_at and last_successful_test_at. asyncpg strictly enforces Python datetime instances for PostgreSQL timestamptz columns ($5, $6). Passing strings directly will raise asyncpg.exceptions.DataError at runtime.

Recommendation: Parse into datetime before binding:

last_tested = datetime.fromisoformat(result["last_tested_at"]) if result.get("last_tested_at") else None
last_success = datetime.fromisoformat(result["last_successful_test_at"]) if result.get("last_successful_test_at") else None

3. Missing failure callback when single key is configured without fallback (services/llm_credential_service.py:398)

chain = _tracked_platform(self._credentials, user_id) if user_settings.get("fallback_to_platform", True) else None
for record in reversed(records):
    ...
    chain = FallbackLLMProvider(primary, chain, _on_primary_failure) if chain else primary

Issue: When fallback_to_platform is false and there is only one key (or on the last record when chain starts as None), chain is assigned directly to primary without wrapping in FallbackLLMProvider. Consequently, _on_primary_failure is never registered. If that credential fails, its status in the database is never updated to FAILED.


4. AnthropicProvider.generate_structured array truncation & placement (providers/llm/__init__.py:58-116)

  • Array Schemas: The regex/slice fallback:
    start = cleaned.find("{")
    end = cleaned.rfind("}")
    return json.loads(cleaned[start:end + 1])
    assumes structured output is always an object {...}. If the target schema is a top-level array [...] (e.g. lists of questions or rubric items), searching for { and } will slice invalid substrings or fail.
  • Architectural Organization: AnthropicProvider is currently defined directly inside providers/llm/__init__.py with local imports inside methods. It should ideally live in its own file (providers/llm/anthropic.py), matching gemini.py, openai.py, and groq.py.

5. is vs == for Enum comparisons (api/dependencies.py:17, api/routes/jobs.py:30,44, api/routes/llm_credentials.py:89)

Using identity comparison is for enum values (principal.principal_type is PrincipalType.ADMIN) is fragile across deserialization boundaries. Use equality == or !=.


6. Frontend UI card re-render overwrites status message (pages/profile.html:281, 296)

In pages/profile.html, testing a key sets statusEl.textContent and immediately calls await renderApiKeysCard(profile). Because renderApiKeysCard replaces container.innerHTML, the status element is instantly destroyed before the user can view the feedback. Updating the DOM row in place or deferring the re-render resolves the flicker.


7. Migration & Schema Redundancies (schema.sql & migrations/)

  • In repositories/postgres/schema.sql, lines 53–57 and 81–82 repeat ALTER TABLE ADD COLUMN IF NOT EXISTS for columns already declared in the CREATE TABLE definitions directly above them.
  • 20261102_02_provider_health.sql repeats column additions already present in 20261102_byok_multi_credentials.sql. Cleaning these up will prevent migration noise.

8. PR Scope & Description

The PR description is currently empty despite including ~2,900 lines of diff that also deprecate /live-sessions/demo and introduce session upgrade endpoints. Adding a brief summary in the PR description will provide helpful context for maintainers and future audits.

@DevSidd2006

Copy link
Copy Markdown
Owner

@MrHarshalBane Thanks for the review, I will check it and do necessary changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants