Fix BIA fallback hit extraction (avoid Untitled results)

.github/copilot-instructions.md

@@ -23,6 +23,14 @@ Example usages of the web python kernel:
 
 The only backends for agent chat are the ri-scale/chat-proxy app in Hypha and the OpenAI API (which is called from the chat-proxy). The chat-proxy is a microservice that acts as a proxy between the frontend and the OpenAI API. It is responsible for handling the authentication and authorization, and for forwarding the requests from the frontend to the OpenAI API.
 
+### Agent architecture guardrails (important)
+
+- Keep `src/pages/AgentPage.tsx` agent-agnostic. Do not add BioImage Finder specific parsing, ranking, or formatting there.
+- Keep all BioImage Finder domain behavior in the BioImage Finder startup script (`scripts/agent_startup_scripts/bioimage_finder_startup_script.py`).
+- If an agent needs compact tool payloads or a structured fallback summary, implement those in that agent's startup script and return them as tool output.
+- Keep chat-proxy app agent-agnostic with only generic methods (`setup`, `chat_completion`, `resolve_url`).
+- Route external archive HTTP calls through `resolve_url` to avoid frontend CORS issues.
+
 ## Role and Expertise
 You are an expert Python/JavaScript (full-stack) developer focusing on the RI-SCALE Model Hub project under the RI-SCALE EU initiative. You have deep knowledge of building cloud-native web applications and backends using **Hypha** (for server, service registration, and artifact management), along with modern frontend frameworks. Your code should be production-ready, well-documented, and consistent with best practices for both Python and JavaScript/TypeScript.
 

README.md

@@ -57,3 +57,10 @@ Artifacts containing an `index.html` file will display an **"Open App"** button
 For branch-safe dev deployments, production-only app IDs, health monitoring, and rollback automation, see:
 
 - `docs/chat-proxy-cicd.md`
+
+## Agent Architecture Principles
+
+- Keep `src/pages/AgentPage.tsx` agent-agnostic. It should orchestrate execution, retries, and generic fallback handling only.
+- Keep BioImage Finder domain logic in `scripts/agent_startup_scripts/bioimage_finder_startup_script.py`.
+- BioImage-specific search result shaping (compact payloads) and dataset fallback summaries must be implemented in the startup script, not in generic frontend orchestration code.
+- Keep chat-proxy app agent-agnostic with a minimal surface (`setup`, `chat_completion`, `resolve_url`).

chat-proxy-app/app.py

@@ -2,7 +2,9 @@
 import logging
 import os
 from typing import Any
+from urllib.parse import urlparse
 
+import httpx
 from hypha_rpc import api
 from openai import AsyncOpenAI
 
@@ -11,6 +13,36 @@
 logger.setLevel(logging.INFO)
 
 _client: AsyncOpenAI | None = None
+_DEFAULT_ALLOWED_HOSTS = "beta.bioimagearchive.org,www.ebi.ac.uk"
+
+
+def _allowed_hosts() -> set[str]:
+    raw_value = os.environ.get("RESOLVE_URL_ALLOWED_HOSTS", _DEFAULT_ALLOWED_HOSTS)
+    hosts = {part.strip().lower() for part in raw_value.split(",") if part.strip()}
+    return hosts
+
+
+def _normalize_headers(headers: dict[str, Any] | None) -> dict[str, str]:
+    normalized: dict[str, str] = {}
+    if not isinstance(headers, dict):
+        return normalized
+    for key, value in headers.items():
+        if not isinstance(key, str):
+            continue
+        if isinstance(value, str):
+            normalized[key] = value
+        elif value is not None:
+            normalized[key] = str(value)
+    return normalized
+
+
+def _error_payload(url: str, status_code: int, error: str) -> dict[str, Any]:
+    return {
+        "ok": False,
+        "status_code": int(status_code),
+        "url": str(url),
+        "error": error,
+    }
 
 
 async def _resolve_openai_key() -> str | None:
@@ -55,11 +87,86 @@ async def setup() -> dict[str, Any]:
     return {"ok": True}
 
 
+async def resolve_url(
+    url: str,
+    method: str = "GET",
+    headers: dict[str, Any] | None = None,
+    timeout: float = 30.0,
+    body: str | dict[str, Any] | list[Any] | None = None,
+    context: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+    try:
+        parsed = urlparse(str(url))
+    except Exception as exp:
+        return _error_payload(str(url), 400, f"Invalid URL: {exp}")
+
+    host = (parsed.hostname or "").lower()
+    if parsed.scheme.lower() != "https":
+        return _error_payload(str(url), 400, "Only https URLs are allowed")
+
+    if host not in _allowed_hosts():
+        return _error_payload(
+            str(url),
+            403,
+            f"Host '{host}' is not allowed by RESOLVE_URL_ALLOWED_HOSTS",
+        )
+
+    method_value = str(method or "GET").upper()
+    if method_value not in {"GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"}:
+        return _error_payload(str(url), 400, f"Unsupported method: {method_value}")
+
+    request_headers = _normalize_headers(headers)
+    request_headers.setdefault("User-Agent", "ri-scale-model-hub-chat-proxy/1.0")
+
+    timeout_seconds = max(1.0, float(timeout))
+
+    request_kwargs: dict[str, Any] = {
+        "method": method_value,
+        "url": str(url),
+        "headers": request_headers,
+    }
+    if body is not None:
+        if isinstance(body, (dict, list)):
+            request_kwargs["json"] = body
+        else:
+            request_kwargs["content"] = str(body)
+
+    try:
+        async with httpx.AsyncClient(
+            timeout=timeout_seconds, follow_redirects=True
+        ) as client:
+            response = await client.request(**request_kwargs)
+    except Exception as exp:
+        logger.warning("resolve_url failed for %s: %s", url, exp)
+        return _error_payload(str(url), 502, str(exp))
+
+    result: dict[str, Any] = {
+        "ok": 200 <= int(response.status_code) < 300,
+        "status_code": int(response.status_code),
+        "url": str(response.url),
+        "headers": dict(response.headers),
+    }
+
+    content_type = response.headers.get("content-type", "")
+    if "application/json" in content_type.lower():
+        try:
+            result["json"] = response.json()
+        except Exception:
+            result["text"] = response.text
+    else:
+        result["text"] = response.text
+
+    if not result["ok"]:
+        result["error"] = f"Upstream returned HTTP {response.status_code}"
+
+    return result
+
+
 async def chat_completion(
     messages: list[dict[str, Any]],
     tools: list[dict[str, Any]] | None = None,
     tool_choice: dict[str, Any] | str | None = None,
-    model: str = "gpt-4-turbo-preview",
+    model: str = "gpt-5-mini",
     context: dict[str, Any] | None = None,
 ) -> dict[str, Any]:
     global _client
@@ -89,5 +196,6 @@ async def chat_completion(
         "config": {"visibility": "public"},
         "setup": setup,
         "chat_completion": chat_completion,
+        "resolve_url": resolve_url,
     }
 )

docs/README.md

@@ -63,9 +63,12 @@ The RI-SCALE agent stack is intentionally simple:
 Current frontend expects chat-proxy to expose:
 
 - `chat_completion(messages, tools, tool_choice, model)`
+- `resolve_url(url, method='GET', headers=None, timeout=30.0)`
 
 The service must be **publicly visible** for anonymous users.
 
+`resolve_url` is used as a CORS-safe relay for selected agent tool HTTP calls (for example, `beta.bioimagearchive.org`) so browser-origin CORS restrictions do not break tool execution.
+
 ---
 
 ## Deploying Chat Proxy (Developer)
@@ -104,6 +107,29 @@ Important notes:
 - It can still happen on deployed domains unless that origin is explicitly allowed by the target server.
 - For reliability, route archive/network fetches through a backend proxy service where possible.
 
+### BioImage Finder Result Quality
+
+For BioImage Archive dataset requests, the BioImage Finder startup script applies a relevance strategy designed for noisy/intermittent beta-index results:
+
+1. Build a brief OR-style query first (for example: `mouse OR tumor`).
+2. If primary query quality is weak or empty, run fallback single-term queries (up to four terms).
+3. Merge unique results from fallback queries and rerank by request-term relevance.
+4. Return the top compact list with accessions/links and a clear beta-index limitation note.
+
+Implementation details live in:
+
+- `scripts/agent_startup_scripts/bioimage_finder_startup_script.py`
+- `docs/bioimage-finder-startup-script.py`
+
+The startup script includes:
+
+- query-term extraction with stopword filtering,
+- relevance scoring over title/description/accession,
+- duplicate-safe result merging across fallback terms,
+- assistant summaries optimized for concise user-facing answers.
+
+To validate quality behavior quickly, run the startup script checks against real API responses and inspect generated summaries for mixed-term prompts (for example, `mouse tumor cancer`).
+
 ### Kernel Logs and Debug Report
 
 Agent chat includes a **Kernel Logs** panel for low-level diagnostics.

docs/bioimage-archive-search-incident-report-2026-02-23.md

@@ -0,0 +1,86 @@
+# BioImage Archive Search Incident Report (for Maintainers)
+
+Date: 2026-02-23  
+Prepared by: RI-SCALE Model Hub team
+
+## 1) Summary
+We observed repeated search failures from the BioImage Archive search endpoints during an end-user query flow. The same session later ended with a model/proxy timeout. 
+
+Current status at the time of this report: direct re-probing of the same endpoint/query patterns returned HTTP 200 consistently, which suggests an intermittent upstream issue rather than a permanent outage.
+
+## 2) User-visible impact
+- User request: "find me 5 mouse tumor datasets"
+- Result during failing run: tool calls repeatedly failed, then final assistant message reported timeout.
+- End-user message shown: "Error from proxy: Request timed out."
+
+## 3) Evidence from failing runtime session (captured logs)
+The following tool calls failed with server-side 500 errors:
+
+- `search_datasets` with query `(mouse OR mice OR murine) AND (tumor OR tumour OR cancer)`  
+  Error text: `Server error '500' for url 'https://beta.bioimagearchive.org/search/search/fts?query=(mouse%20OR%20mice%20OR%20murine)%20AND%20(tumor%20OR%20tumour%20OR%20cancer)'`
+
+- `search_datasets` with query `mouse OR mice OR murine OR tumor OR tumour OR cancer`  
+  Error text: `Server error '500' for url 'https://beta.bioimagearchive.org/search/search/fts?query=mouse%20OR%20mice%20OR%20murine%20OR%20tumor%20OR%20tumour%20OR%20cancer'`
+
+- `search_datasets` with query `mouse OR tumor`  
+  Error text: `Server error '500' for url 'https://beta.bioimagearchive.org/search/search/fts?query=mouse%20OR%20tumor'`
+
+- `search_images` with query `mouse AND tumor`  
+  Error text: `Server error '500' for url 'https://beta.bioimagearchive.org/search/search/fts/image?query=mouse%20AND%20tumor'`
+
+Additional session outcome:
+- Proxy/model loop eventually returned: `{"error": "Request timed out."}`
+
+## 4) Independent direct endpoint probe (performed after incident)
+To verify endpoint health, we queried the same URL patterns directly from the client environment (outside browser CORS path).
+
+Probe window (UTC): 2026-02-23T12:52:32Z to 2026-02-23T12:52:41Z  
+Total requests: 15  
+HTTP status distribution: 15x 200
+
+### Probe results (timestamped)
+
+| timestamp_utc | attempt | endpoint | query_label | http_code |
+|---|---:|---|---|---:|
+| 2026-02-23T12:52:32Z | 1 | fts | q_complex | 200 |
+| 2026-02-23T12:52:33Z | 1 | fts | q_or_long | 200 |
+| 2026-02-23T12:52:34Z | 1 | fts | q_short | 200 |
+| 2026-02-23T12:52:34Z | 1 | fts/image | q_image_and | 200 |
+| 2026-02-23T12:52:35Z | 1 | fts/image | q_image_or | 200 |
+| 2026-02-23T12:52:36Z | 2 | fts | q_complex | 200 |
+| 2026-02-23T12:52:36Z | 2 | fts | q_or_long | 200 |
+| 2026-02-23T12:52:37Z | 2 | fts | q_short | 200 |
+| 2026-02-23T12:52:38Z | 2 | fts/image | q_image_and | 200 |
+| 2026-02-23T12:52:38Z | 2 | fts/image | q_image_or | 200 |
+| 2026-02-23T12:52:39Z | 3 | fts | q_complex | 200 |
+| 2026-02-23T12:52:39Z | 3 | fts | q_or_long | 200 |
+| 2026-02-23T12:52:40Z | 3 | fts | q_short | 200 |
+| 2026-02-23T12:52:41Z | 3 | fts/image | q_image_and | 200 |
+| 2026-02-23T12:52:41Z | 3 | fts/image | q_image_or | 200 |
+
+One sampled successful response during probe:
+- URL: `https://beta.bioimagearchive.org/search/search/fts?query=mouse%20OR%20tumor`
+- Status: `HTTP/2 200`
+- Body prefix: `{"hits":{"total":{"value":0,"relation":"eq"},...}` (valid JSON payload)
+
+## 5) Assessment
+Based on this evidence:
+- The failure is real (multiple 500 responses during user session across both dataset and image search endpoints).
+- The issue appears intermittent/transient (subsequent direct probes returned stable 200 responses).
+- This pattern is consistent with temporary upstream backend/index/gateway instability rather than a persistent schema/query-format issue.
+
+## 6) Suggested maintainer checks (respectfully suggested)
+- Review server logs for the failing timestamps around the user session (especially 5xx bursts on `/search/search/fts` and `/search/search/fts/image`).
+- Check backend search cluster health, queue saturation, and timeouts.
+- Check gateway/load balancer/WAF behavior for transient 5xx responses.
+- Confirm whether any rolling deploy/reindex/maintenance occurred during the failure window.
+
+## 7) Repro URLs used
+- `https://beta.bioimagearchive.org/search/search/fts?query=(mouse%20OR%20mice%20OR%20murine)%20AND%20(tumor%20OR%20tumour%20OR%20cancer)`
+- `https://beta.bioimagearchive.org/search/search/fts?query=mouse%20OR%20mice%20OR%20murine%20OR%20tumor%20OR%20tumour%20OR%20cancer`
+- `https://beta.bioimagearchive.org/search/search/fts?query=mouse%20OR%20tumor`
+- `https://beta.bioimagearchive.org/search/search/fts/image?query=mouse%20AND%20tumor`
+- `https://beta.bioimagearchive.org/search/search/fts/image?query=mouse%20OR%20tumor`
+
+---
+We appreciate the BioImage Archive team’s support and understand intermittent issues can happen in production systems. We hope this report is useful and are happy to provide any additional logs or run further directed probes if helpful.