feat: add optional you.com search integration - #22
Open
mouse-value-add wants to merge 1 commit into
Open
Conversation
Adds an opt-in third gather angle to search_structured: a You.com web search that surfaces candidate repos from where they live in discourse (awesome lists, comparisons, blog posts) β a blind spot shared by the keyword and topic queries, which both hit the GitHub search index. - Off by default: only runs when YDC_API_KEY is set; DEEPGIT_YOUCOM_SEARCH=0 disables it even with a key. No key -> byte-identical behavior. - Zero LLM tokens, zero new dependencies (httpx already used for GitHub). - Web-surfaced slugs are hydrated via the existing GitHubGraphQL client (repo:<owner>/<name> qualifier) so every candidate is judged on the same metadata + README + root tree evidence as the other angles. - check_setup() now reports youcom_web_search status. - Fails silently to GitHub-only results on HTTP errors/timeouts, matching the optional semantic-recall layer's contract. - Offline unit tests for slug extraction, gating, and error paths.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An opt-in You.com web-search gather lane for
search_structured. DeepGit's keyword and star-sorted topic queries both hit the GitHub search index, so they share a blind spot: repos that keyword search buries, or whose canonical descriptions omit the terms a developer would actually use (the same gap the topic sweep and the sparse-result retry already work around). A general web search sees those repos where they actually live in discourse β awesome lists, "X vs Y" comparisons, blog posts.When enabled, the lane:
owner/nameslugs fromgithub.com/...links in result URLs, titles, descriptions, and snippets (site pages likegithub.com/aboutare filtered out)GitHubGraphQLclient (repo:<owner>/<name>qualifier) so every web-surfaced candidate carries the same metadata + README + root tree evidence as the other anglesOpt-in by default-off
YDC_API_KEYβ the lane never runs; behavior is byte-identical to todayDEEPGIT_YOUCOM_SEARCH=0β hard opt-out even with a keyDEEPGIT_YOUCOM_MAX_QUERIESβ tune web queries per search (default 2)check_setup()now reports ayoucom_web_searchblock so users can confirm the lane stateThis follows the repo's existing pattern for optional capability: the
[semantic]extra andenable_code_qualityare both opt-in, and this lane adds no new dependency (httpx is already used for the GitHub client).What changed
src/deepgit/search/youdotcom.py(new)src/deepgit/search/__init__.py(new)src/deepgit/pipeline.pysrc/deepgit/config.pyyoucom_api_key,youcom_search,youcom_max_queriessrc/deepgit/mcp_server.pycheck_setup()reports the lanetests/test_youdotcom.py(new)README.md,.env.example,pyproject.tomlSetup
Validation
python -m pytest tests/β 16 passed (extraction, gating, silent-failure, max-slug cap)ruff check src/deepgit/search/ tests/ src/deepgit/pipeline.pyβ clean;ruff formatappliedmypy src/deepgit/search/youdotcom.pyβ clean (config/pipeline have pre-existing errors on main, unchanged by this PR)check_setup()verified in all three states (no key / key / opt-out)results.webβ slug-extraction chain returns real repo slugs from real search resultsyoucom_enabled()isFalseandgather_web_recordsis never invoked (asserted in tests)Happy to adjust the shape if you'd rather have this behind a
[youcom]extra, a different settings name, or as a gather-layer plugin point instead of inline insearch_structured.