Skip to content

feat: add optional you.com search integration - #22

Open
mouse-value-add wants to merge 1 commit into
zamalali:mainfrom
mouse-value-add:feat/youcom-search-integration
Open

feat: add optional you.com search integration#22
mouse-value-add wants to merge 1 commit into
zamalali:mainfrom
mouse-value-add:feat/youcom-search-integration

Conversation

@mouse-value-add

Copy link
Copy Markdown

What this adds

An opt-in You.com web-search gather lane for search_structured. DeepGit's keyword and star-sorted topic queries both hit the GitHub search index, so they share a blind spot: repos that keyword search buries, or whose canonical descriptions omit the terms a developer would actually use (the same gap the topic sweep and the sparse-result retry already work around). A general web search sees those repos where they actually live in discourse β€” awesome lists, "X vs Y" comparisons, blog posts.

When enabled, the lane:

  1. Queries the You.com Search API (0 LLM tokens) with the intent plus one keyword query
  2. Extracts owner/name slugs from github.com/... links in result URLs, titles, descriptions, and snippets (site pages like github.com/about are filtered out)
  3. Hydrates those slugs through the existing GitHubGraphQL client (repo:<owner>/<name> qualifier) so every web-surfaced candidate carries the same metadata + README + root tree evidence as the other angles
  4. Folds the new candidates into the pool before the zero-token prefilter β€” the judge and adaptive controller are untouched

Opt-in by default-off

  • No YDC_API_KEY β†’ the lane never runs; behavior is byte-identical to today
  • DEEPGIT_YOUCOM_SEARCH=0 β†’ hard opt-out even with a key
  • DEEPGIT_YOUCOM_MAX_QUERIES β†’ tune web queries per search (default 2)
  • HTTP errors / timeouts / no slugs found β†’ returns no candidates and the pipeline continues on GitHub-only results (same failure contract as the optional semantic-recall layer)
  • check_setup() now reports a youcom_web_search block so users can confirm the lane state

This follows the repo's existing pattern for optional capability: the [semantic] extra and enable_code_quality are both opt-in, and this lane adds no new dependency (httpx is already used for the GitHub client).

What changed

File Change
src/deepgit/search/youdotcom.py (new) Search API client + slug extraction + GraphQL hydration
src/deepgit/search/__init__.py (new) Package exports
src/deepgit/pipeline.py Lane wired in at step 2a (between gather and semantic layer), fully gated
src/deepgit/config.py 3 settings: youcom_api_key, youcom_search, youcom_max_queries
src/deepgit/mcp_server.py check_setup() reports the lane
tests/test_youdotcom.py (new) 16 offline unit tests (extraction, gating, error paths)
README.md, .env.example, pyproject.toml Docs + testpaths

Setup

export YDC_API_KEY=***   # https://you.com/platform/api-keys
# optional: DEEPGIT_YOUCOM_SEARCH=0, DEEPGIT_YOUCOM_MAX_QUERIES=2

Validation

  • python -m pytest tests/ β†’ 16 passed (extraction, gating, silent-failure, max-slug cap)
  • ruff check src/deepgit/search/ tests/ src/deepgit/pipeline.py β†’ clean; ruff format applied
  • mypy src/deepgit/search/youdotcom.py β†’ clean (config/pipeline have pre-existing errors on main, unchanged by this PR)
  • check_setup() verified in all three states (no key / key / opt-out)
  • Live check against You.com's keyless search surface confirmed the results.web β†’ slug-extraction chain returns real repo slugs from real search results
  • No key present: youcom_enabled() is False and gather_web_records is never invoked (asserted in tests)

Happy to adjust the shape if you'd rather have this behind a [youcom] extra, a different settings name, or as a gather-layer plugin point instead of inline in search_structured.

Adds an opt-in third gather angle to search_structured: a You.com web
search that surfaces candidate repos from where they live in discourse
(awesome lists, comparisons, blog posts) β€” a blind spot shared by the
keyword and topic queries, which both hit the GitHub search index.

- Off by default: only runs when YDC_API_KEY is set; DEEPGIT_YOUCOM_SEARCH=0
  disables it even with a key. No key -> byte-identical behavior.
- Zero LLM tokens, zero new dependencies (httpx already used for GitHub).
- Web-surfaced slugs are hydrated via the existing GitHubGraphQL client
  (repo:<owner>/<name> qualifier) so every candidate is judged on the same
  metadata + README + root tree evidence as the other angles.
- check_setup() now reports youcom_web_search status.
- Fails silently to GitHub-only results on HTTP errors/timeouts, matching
  the optional semantic-recall layer's contract.
- Offline unit tests for slug extraction, gating, and error paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant