[krishika08]

Summary

Release candidate for RepoNeedle 1.2.0. The package version is intentionally still 1.1.0; the version bump, tag and PyPI publication are separate follow-up steps after review.

This PR adds the native MCP server, machine-readable CLI output with anchor files, indexing diagnostics, and an opt-in default for the experimental abstention heuristic. It also fixes a security issue in repository indexing, restores a behaviour from the published 1.1.0 wheel, hardens CI and the release workflow, and reconciles documentation and benchmark claims with the code.

⚠️ Behaviour change for users upgrading from 1.1.x

Experimental abstention is now disabled by default. The heuristic is uncalibrated: it correctly abstained on 0 of 82 no-gold queries in the ARB v2 selective split. Out-of-scope queries now return ranked results unless the caller opts in:

• Python: BudgetConfig(enable_experimental_abstention=True) or retrieve(..., enable_experimental_abstention=True)
• CLI: --enable-experimental-abstention
• MCP: "enable_experimental_abstention": true

Empty repositories still return abstained=True. task_type="selective" does not enable the heuristic.

🔒 Security fix: indexing could read files outside the repository through links

RepoNeedleEngine.from_directory() previously opened files reached through symlinks (and, on Windows, NTFS junctions) without a containment check. A cloned repository containing, for example, docs/notes.md → ~/.aws/credentials would have that file indexed, and its contents could be returned by retrieval, including through the MCP reponeedle_retrieve tool. Reproduced against the released 1.1.0 code with a junction and a sentinel file: the sentinel appeared in rendered_context.

Fix (reponeedle/engine.py):

• Resolve the canonical repository root once.
• Resolve each candidate with os.path.realpath(strict=True) before opening it; broken links and loops are recorded as unreadable.
• Check containment by path component (os.path.commonpath on case-normalised paths), not by string prefix.
• Open only the validated resolved path, never the link.
• Skip files resolving outside the root without reading the target; record only the in-repository link path in indexing_diagnostics.symlink_outside_repository_paths (reason symlink_outside_repository), plus a count-only warning.
• Links resolving inside the repository are still indexed.

CLI anchors now apply the same canonical containment check. CLI and MCP path validation are component-aware: legitimate names such as ..cache/x.py are accepted, and real ../ traversal is still rejected.

Changes

MCP server (new, optional extra mcp)

• reponeedle-mcp / python -m reponeedle.mcp, STDIO transport, with tools reponeedle_retrieve, reponeedle_change_impact, reponeedle_inspect.
• Indexes once at startup and reuses the engine; path arguments are confined to the repository root.
• serverInfo.version now reports RepoNeedle's version (previously the MCP SDK's).
• Without the extra, the core package works and reponeedle-mcp prints installation instructions.

CLI

• --json (single JSON document on stdout; diagnostics on stderr only), --include-context, --anchors, --enable-experimental-abstention, --max-file-size-bytes.

Python API (backward compatible)

• retrieve(anchor_files=...), Query(anchor_files=...), from_directory(max_file_size_bytes=...).
• engine.indexing_diagnostics and RetrievalResult.abstention.
• New exports: AbstentionDiagnostics, IndexingDiagnostics, DEFAULT_MAX_FILE_SIZE_BYTES.

Compatibility with 1.1.0 (main)

• main has been merged into this branch (0adfbba). The change_impact task-reporting logic (router-classified task for AUTO queries) is identical to main; this PR adds regression tests for it. Reporting only; ranking unchanged.
• The optional api extra (fastapi, httpx, uvicorn) for the root-level demo api.py is kept. No module inside the reponeedle package imports these packages.
• Correction: an earlier version of this description and of the release audit said the published 1.1.0 wheel was built from source that differed from the repository. That was wrong: it came from a stale local origin/main. The wheel matches tag v1.1.0.

CI and release workflow

• pytest-asyncio added to the dev extra.
• New CI job mcp-security-linux (Ubuntu, Python 3.12, .[dev,mcp]) runs the MCP, symlink-security and compatibility suites with REPONEEDLE_FAIL_ON_SKIP=1: any skipped test fails the job. It also runs a live MCP STDIO smoke test that checks serverInfo.
• Release workflow: a new validate job (read-only, no publishing credentials) verifies that the tag, pyproject.toml and reponeedle.__version__ match, installs .[dev,mcp,api], runs the full suite and the strict MCP/security suites; build depends on it and checks the built artifact versions; publish depends on both and is the only job with id-token: write.

Documentation and benchmark integrity

• README restructured (quickstart, release-status table, clearly separated benchmark kinds).
• Historical ARB v2 results kept as published and labelled with their engine provenance (commit 8e00425, 2026-10-28). Current-engine official-harness results, a supplemental whole-file evaluation, and experimental studies are reported separately and labelled as not official (docs/BENCHMARK_CONFIGURATION.md, reports/experimental/).
• Corrected inaccurate claims: synthetic "100% abstention accuracy", "calibrated abstention", "100% test coverage", threshold and score ranges, and the comment2context root-cause section (structured JSON PR-review queries; verified fusion weights 0.40/0.40/0.20 with a zero graph term by default; removed the unsupported "BM25 ≥ 0.85" recommendation).
• Experimental results included for reproducibility: documentation filter not validated; lexical optimisation not validated (query-term-frequency handling identified as the strongest candidate cause; no production change).
• Release audit and proposed changelog: reports/release_audit_1.2.0/.

Testing

Check	Result
Full suite after merging main, Python 3.12 (Windows, mcp + api extras)	481 passed, 7 skipped (all 7: file-symlink creation not permitted on Windows)
Full suite, Python 3.11 (Windows, no mcp)	437 passed, 23 skipped (MCP not installed; file symlinks)
All changed .py files compiled with Python 3.11 (pre-3.12 f-string grammar)	38/38
Clean install from built wheel, Python 3.11, base only	38/38 functional checks (CLI JSON, anchors, traversal rejection, abstention default and opt-ins, size limits, junction escape blocked, friendly MCP-missing error)
Clean install from built wheel, Python 3.12, [mcp]	49/49 checks, including a live MCP STDIO session (exact tool set and arguments, path-escape rejection, serverInfo version)
python -m build + twine check --strict	PASSED (wheel contains only reponeedle/ incl. MCP; runtime dependencies unchanged)
Strict no-skip mode (REPONEEDLE_FAIL_ON_SKIP=1)	Verified: exit code 1 when any test is skipped
Release version check	Verified: a mismatched tag (v1.2.0 vs 1.1.0) is refused
Historical benchmark result files	Unchanged (SHA-256 verified)

❗ Remaining verification required on Linux

The six file-symlink security tests in tests/test_indexing_security.py and the existing MCP symlink-escape test could not be executed locally: Windows here does not permit symlink creation, and no Linux environment (WSL/Docker) was available. The indexing containment path itself was verified empirically on Windows using NTFS junctions.

This PR's mcp-security-linux job must pass with zero skipped tests before the release blockers can be considered fully resolved.

Not included

• No version bump (still 1.1.0), no tag, no PyPI or Hugging Face publication.
• After merge: bump to 1.2.0, update the pre-release wording in the docs ("planned for 1.2.0", "MCP from source only"), add CHANGELOG.md from reports/release_audit_1.2.0/PROPOSED_CHANGELOG_1.2.0.md, then tag v1.2.0 and let the release workflow publish.

🤖 Generated with Claude Code