deploy: prepare Hugging Face and demo API for 1.2.0

HF_RELEASE_CHECKLIST.md

@@ -1,7 +1,7 @@
 # HUGGING FACE & OPEN-SOURCE RELEASE CHECKLIST: REPONEEDLE
 
 > [!NOTE]
-> **Historical record (2026-10-28) of the v0.1.0 release checklist.** It has not been updated for later versions. The Hugging Face Space in `hf_artifacts/space/` bundles the 1.1.0 engine, in which the uncalibrated abstention heuristic is always on.
+> **Historical record (2026-10-28) of the v0.1.0 release checklist.** It has not been updated for later versions. When this note was added (2026-11-10), the Hugging Face Space in `hf_artifacts/space/` bundled the 1.1.0 engine, in which the uncalibrated abstention heuristic is always on. The Space bundle has since been synchronised with RepoNeedle 1.2.0, where the heuristic is opt-in; see `docs/STATIC_PLAYGROUND_DEPLOYMENT.md`.
 
 **Version**: `0.1.0` (Research Preview)  
 **Date**: 2026-10-28  

api.py

@@ -5,6 +5,7 @@
 """
 from __future__ import annotations
 
+import logging
 import sys
 import time
 from functools import lru_cache
@@ -22,19 +23,24 @@
 
 # app.py handles Gradio as optional; it can be imported without installing Gradio.
 from app import CORPORA  # noqa: E402
+import reponeedle  # noqa: E402
 from reponeedle.engine import RepoNeedleEngine  # noqa: E402
 from reponeedle.models import Query, TaskType  # noqa: E402
 from reponeedle.core.lexical import BM25Index  # noqa: E402
 from reponeedle.core.graph import RepoGraph  # noqa: E402
 
+# Single source of truth: the version of the bundled engine this API actually imports.
+ENGINE_VERSION = reponeedle.__version__
+
+logger = logging.getLogger("reponeedle.demo_api")
 
 app = FastAPI(
     title="RepoNeedle Demo API",
     description=(
-        "Browser-facing API for the RepoNeedle V1.1.0 demonstration. "
+        f"Browser-facing API for the RepoNeedle {ENGINE_VERSION} demonstration. "
         "Uses embedded sample snapshots of Flask and Requests."
     ),
-    version="1.1.0",
+    version=ENGINE_VERSION,
 )
 
 # Hugging Face Static Spaces are served from a different origin.
@@ -56,6 +62,9 @@ class RetrieveRequest(BaseModel):
     budget_tokens: int = Field(default=4000, ge=500, le=8000)
     seed_files: list[str] = Field(default_factory=list, max_length=10)
     query: str = Field(min_length=2, max_length=3000)
+    # Experimental, uncalibrated out-of-scope abstention heuristic. Off unless the caller opts in
+    # (the RepoNeedle 1.2.0 default).
+    enable_experimental_abstention: bool = False
 
 
 @lru_cache(maxsize=2)
@@ -78,7 +87,7 @@ def get_baselines(repo_name: str):
 def root():
     return {
         "name": "RepoNeedle Demo API",
-        "version": "1.1.0",
+        "version": ENGINE_VERSION,
         "health": "/health",
         "docs": "/docs",
         "disclosure": "Embedded demonstration snapshots only; not full upstream repositories.",
@@ -87,7 +96,7 @@ def root():
 
 @app.get("/health")
 def health():
-    return {"status": "ok", "service": "reponeedle-demo-api", "version": "1.1.0"}
+    return {"status": "ok", "service": "reponeedle-demo-api", "version": ENGINE_VERSION}
 
 
 @app.post("/api/retrieve")
@@ -112,7 +121,11 @@ def retrieve_context(payload: RetrieveRequest):
         )
 
         started = time.perf_counter()
-        result = engine.retrieve(query=query_obj, ranker=payload.ranker)
+        result = engine.retrieve(
+            query=query_obj,
+            ranker=payload.ranker,
+            enable_experimental_abstention=payload.enable_experimental_abstention,
+        )
         elapsed_ms = (time.perf_counter() - started) * 1000
 
         selected = []
@@ -152,6 +165,14 @@ def retrieve_context(payload: RetrieveRequest):
         else:
             ranker_note = "Legacy ranking is active (the backwards-compatible default ranker)."
 
+        diag = result.abstention
+        abstention = {
+            "enabled": bool(diag.enabled) if diag else bool(payload.enable_experimental_abstention),
+            "status": diag.status if diag else ("evaluated" if payload.enable_experimental_abstention else "disabled"),
+            "experimental": True,
+            "calibrated": False,
+        }
+
         return {
             "query": query_text,
             "repository": payload.repo_name,
@@ -169,9 +190,12 @@ def retrieve_context(payload: RetrieveRequest):
             "selected_files": selected,
             "comparison": comparison,
             "preview": selected[0] if selected else None,
+            "abstention": abstention,
         }
     except HTTPException:
         raise
     except Exception as exc:
-        # Do not return stack traces or local filesystem details to public callers.
-        raise HTTPException(status_code=500, detail=f"Retrieval failed: {type(exc).__name__}: {exc}") from exc
+        # Do not return exception messages, stack traces or local filesystem details to public
+        # callers; keep the details in the server log only.
+        logger.exception("Demo retrieval failed")
+        raise HTTPException(status_code=500, detail="Retrieval failed due to an internal error.") from exc

docs/STATIC_PLAYGROUND_DEPLOYMENT.md

@@ -1,16 +1,62 @@
-# RepoNeedle V1.1.0 public demo deployment
+# RepoNeedle 1.2.0 public demo deployment
 
-This document describes the public demo deployment. It does not change the installation path for the core Python package.
+This document describes the public demo deployment. It does not change the installation path for the core Python package (`pip install reponeedle`).
 
 ## Components
 
 | Component | Location | Purpose |
 |---|---|---|
 | Core engine | `reponeedle/` | CPU-first retrieval library and CLI |
-| Static frontend | `hf_artifacts/static_space/` | Browser UI hosted by Hugging Face Static Spaces |
-| Demo API | `api.py` | FastAPI wrapper exposing retrieval for the embedded demo corpora |
-| Demo snapshots and bundled UI engine | `hf_artifacts/space/` | Small Flask and Requests snapshots and synchronized RepoNeedle package used by the API |
-| Dataset | `hf_artifacts/dataset/` | Lightweight synthetic demonstration benchmark |
+| Static frontend | `hf_artifacts/static_space/` | Browser UI hosted by a Hugging Face **Static** Space |
+| Demo API | `api.py` | FastAPI wrapper exposing retrieval for the embedded demo corpora (hosted on Render) |
+| Demo snapshots and bundled engine | `hf_artifacts/space/` | Small Flask and Requests snapshots (`app.py`) and the bundled `reponeedle` package used by the API and the Gradio Space |
+| Gradio Space | `hf_artifacts/space/` | Self-contained Gradio Playground that runs the bundled engine in-process |
+| Dataset | `hf_artifacts/dataset/` | Lightweight synthetic demonstration benchmark (unchanged in 1.2.0) |
+
+### Hugging Face repositories
+
+| Repository | Type | Source in this repository |
+|---|---|---|
+| `Krishikax/RepoNeedle-Playground` | Space, **static** SDK (frontend) | README badge; `hf_artifacts/static_space/` |
+| `RepoNeedle/RepoNeedle-Playground` | Space, **Gradio** SDK | README badge; default target of `scripts/prepare_hf_release.py` (`--org RepoNeedle`) |
+| `RepoNeedle/RepoNeedle-Bench` | Dataset | README badge; `hf_artifacts/dataset/`; default target of `scripts/prepare_hf_release.py` |
+
+Both Spaces are named `RepoNeedle-Playground` but live in different namespaces and use different SDKs. Never publish one directory to the other's repository.
+
+### How the pieces fit together
+
+```
+Static Space (index.html) ──HTTPS──► Render API (api.py) ──imports──► hf_artifacts/space/reponeedle (bundled engine)
+                                                         └─imports──► hf_artifacts/space/app.py (embedded Flask/Requests snapshots)
+Gradio Space (app.py) ──in-process──► hf_artifacts/space/reponeedle (bundled engine)
+```
+
+- The static Space runs no Python. It calls the API URL shown in its **Backend API URL** field (default `https://reponeedle.onrender.com`).
+- `api.py` puts `hf_artifacts/space/` first on `sys.path`, so it runs the **bundled** engine, not the copy that `requirements-api.txt` installs from the repository root.
+- The bundled engine is an exact copy of `reponeedle/` **except** the optional `reponeedle/mcp/` package, which the demo does not need. `tests/test_hf_space_bundle.py` fails if the two diverge, and `scripts/prepare_hf_release.py` refuses to publish a diverged bundle. After changing `reponeedle/`, copy the changed files into `hf_artifacts/space/reponeedle/`.
+- The API's reported version is read from the imported engine's `reponeedle.__version__`, so it always reflects the code actually running.
+
+## Deployment sequence for 1.2.0
+
+The order is API first, then the frontends. The 1.2.0 static UI also works against a 1.1.0 API: the older API ignores the new opt-in field, and its heuristic is always on.
+
+1. **Merge to `main`** once CI passes.
+2. **Render API.** Render builds from `main`. If auto-deploy is enabled for the service, merging starts a deploy; otherwise start a manual deploy in the Render dashboard. Then verify:
+   - `GET https://YOUR-SERVICE.onrender.com/health` returns `{"status":"ok","service":"reponeedle-demo-api","version":"1.2.0"}`;
+   - `GET /` reports `"version": "1.2.0"`.
+3. **Static Space.** Upload `README.md` and `index.html` from `hf_artifacts/static_space/` to the static Space root. This is a manual step; the release script does not publish the static Space. Verify that the page shows `PLAYGROUND · V1.2.0` and that **Connect API** reports `Connected · RepoNeedle API 1.2.0`.
+4. **Gradio Space** (if it is maintained). Run the preflight, then publish the Space only:
+   ```bash
+   python scripts/prepare_hf_release.py --check-only --space-only --org <namespace>
+   python scripts/prepare_hf_release.py --space-only --org <namespace>
+   ```
+   - `--space-only` does not upload the dataset.
+   - The script checks that the bundled engine matches the root package and `pyproject.toml`, never prints the token, and verifies that the Space's head is the uploaded commit.
+   - It refuses to upload into an existing Space whose SDK is not `gradio`, such as the static frontend Space. The Gradio and static Spaces must be different repositories.
+   - The Space installs Gradio `sdk_version` 4.29.0 from its README front matter. `requirements.txt` pins `fastapi<0.113`, which keeps Starlette at 0.38.x: Gradio 4.x cannot serve its page with Starlette 1.x.
+5. **Dataset.** No upload is needed; the demonstration dataset is unchanged in 1.2.0.
+
+Do not describe a remote Space or API as upgraded until its live version has been checked as above.
 
 ## Deploy the API on Render
 
@@ -29,7 +75,7 @@ The API imports the embedded corpora and engine from `hf_artifacts/space/`. Do n
 Verify `https://YOUR-SERVICE.onrender.com/health`. A healthy response is:
 
 ```json
-{"status":"ok","service":"reponeedle-demo-api","version":"1.1.0"}
+{"status":"ok","service":"reponeedle-demo-api","version":"1.2.0"}
 ```
 
 ## Deploy the frontend on Hugging Face
@@ -40,22 +86,35 @@ The UI defaults to `https://reponeedle.onrender.com`. Users can change the Backe
 
 ## API contract
 
-- `GET /` returns service metadata and a scope disclosure.
-- `GET /health` is the health check.
-- `POST /api/retrieve` accepts `repo_name`, `ranker`, `task_mode`, `budget_tokens`, `seed_files`, and `query`.
+- `GET /` returns service metadata (including `version`) and a scope disclosure.
+- `GET /health` is the health check: `{"status": "ok", "service": "reponeedle-demo-api", "version": "<engine version>"}`.
+- `POST /api/retrieve` accepts:
+  - `repo_name`, `ranker`, `task_mode`, `budget_tokens`, `seed_files` and `query`;
+  - optional `enable_experimental_abstention` (boolean, default `false`), new in 1.2.0.
+- Response fields are unchanged from 1.1.0, plus an additive `abstention` object: `{"enabled", "status", "experimental": true, "calibrated": false}`. `status` is `"disabled"` by default and `"evaluated"` when opted in.
 - Supported demo repositories: `pallets/flask`, `psf/requests`.
 - Supported rankers: `legacy` (default), `change_impact`.
-- The API validates query length, budget limits, repository choice, and seed count using Pydantic.
+- The API validates query length, budget limits, repository choice, seed count and the abstention flag type using Pydantic.
+- Internal errors return HTTP 500 with a generic message. Exception details are written to the server log only and are never returned to callers.
+
+## Experimental abstention (1.2.0 default: off)
+
+- In 1.2.0 the out-of-scope abstention heuristic is **disabled by default**. The API, the static UI and the Gradio Space return ranked context for every query unless the caller opts in.
+- The static UI and the Gradio Space have an unchecked **Experimental abstention** control.
+- Choosing the `selective` task does not enable abstention.
+- The heuristic is experimental and **uncalibrated**: on the ARB v2 selective split it correctly abstained on 0 of 82 out-of-scope queries. An abstention on a demo query is a diagnostic, not a reliable out-of-scope signal. See `KNOWN_LIMITATIONS.md`.
+- In 1.1.0 (the previous demo) the heuristic was always on.
 
 ## Limitations and security boundaries
 
 - This is a public, unauthenticated demonstration API. Do not send secrets, proprietary source code, or personal data.
 - CORS is open so the public Hugging Face frontend can call the API. This is not an authentication or abuse-prevention control.
 - The API only retrieves from the two embedded demonstration snapshots. It does not index arbitrary repositories or expose a general upload endpoint.
+- The demos rank in-memory documents (`RepoNeedleEngine.from_documents`). They do not index a directory, so they do not exercise 1.2.0's filesystem indexing safeguards (link containment, directory-link traversal limits) or show indexing diagnostics.
 - The API does not persist visitor queries. Its in-memory caches reset when the service restarts.
 - Free hosting may sleep after inactivity and has limited CPU/RAM. Cold starts and request latency can vary.
 - The dataset is a small synthetic demo dataset, not the complete Agent Retrieval Bench v2 corpus.
-- Selective abstention is experimental and should not be presented as reliable; see `KNOWN_LIMITATIONS.md`.
+- Demo results are not benchmark results; see the README for the documented evaluations and their qualifications.
 
 ## Local development and tests
 
@@ -66,4 +125,6 @@ pip install -e ".[dev,api]"
 pytest -q
 ```
 
-The API tests use FastAPI's `TestClient` and the public-demo corpus shipped with the repository.
+- The API tests (`tests/test_public_api.py`) use FastAPI's `TestClient` and the demo corpora shipped with the repository.
+- `tests/test_hf_space_bundle.py` checks the bundled engine against the root package and smoke-tests the Gradio app without Gradio installed.
+- It also checks the release script's safeguards using a fake Hub client, so nothing is uploaded.

hf_artifacts/space/README.md

@@ -10,16 +10,22 @@ pinned: false
 license: mit
 ---
 
-# 🪡 RepoNeedle Playground (V1.1.0)
+# 🪡 RepoNeedle Playground (V1.2.0)
 
 Interactive demonstration of **RepoNeedle**: Token-Budget-Aware, Workflow-Adaptive Repository Context Retrieval for AI Coding Agents.
 
 > ⚠️ **Methodological Disclosure**: This Playground uses embedded demonstration repository snapshots. It is an interactive demonstration, not a reproduction of the complete ARB v2 evaluation. The embedded corpora are lightweight demonstration slices, not full upstream repository checkouts.
 
-### Key Capabilities in V1.1.0:
+### What this Playground demonstrates (bundled RepoNeedle 1.2.0 engine):
 - **Modular Ranking Engine Selection**: Choose between the default `legacy` ranker (V1.0 backwards-compatible multi-signal routing) and the opt-in `change_impact` ranking engine.
 - **Change-Impact Graph Propagation**: When seed / anchor files are provided, the engine computes seed-relative relationship scoring across static imports, symbols, and calls with Seed File Suppression (SFS).
 - **Seedless Change-Impact Mode**: When run without seed files, the interface clarifies that lexical/semantic scoring and Test Intent Priors remain active while seed-relative relationship boosts and suppression are inactive.
 - **Workflow-Adaptive Routing**: Automatic intent classification for `trace2code`, `code2test`, `edit2ripple`, and `comment2context`.
 - **CUD-Pack Context Optimization**: Deterministic budget enforcement ($C_{\text{actual}} \le B_{\text{configured}}$) across representation tiers (`FULL`, `FOCUS_SPAN`, `SKELETON`).
 - **Multi-System Baseline Comparison**: Side-by-side comparison with classical BM25 and RepoMap PageRank under identical context constraints.
+- **Experimental Abstention (opt-in, off by default)**: In 1.2.0 the out-of-scope abstention heuristic is disabled by default, so ranked context is returned for every query. An unchecked **Enable experimental abstention** box lets you opt in. The heuristic is experimental and uncalibrated: on the ARB v2 selective split it correctly abstained on 0 of 82 out-of-scope queries.
+
+### Scope of the demonstration
+
+- The Playground ranks small embedded document snapshots held in memory. It does not index a directory, so it does not exercise 1.2.0's filesystem-level indexing safeguards (link containment, directory-link traversal limits) or report indexing diagnostics.
+- The CLI JSON output, `--anchors` validation and the MCP server added in 1.2.0 are not part of this Space. See the [RepoNeedle repository](https://github.com/krishika08/RepoNeedle) for those features and for the documented benchmark results and limitations.