Cloud sweep: package doctests in CI, wheel ships skills, pgvector 0.5 fix, LanceDB native hybrid, native async registry, setup skill - #25
Merged
Conversation
CI invoked pytest with no path, so testpaths=["tests"] meant none of the ~290 doctest lines under vd/ ever ran. Add `vd` to testpaths and a root conftest.py that skips backend modules whose optional SDK is missing (and misc/ demo scripts), so collection never aborts on an ImportError. Two doctests in vd/text.py had wrong expected output (clean_text with remove_punctuation, extract_metadata char_count); the code was right. Fixes #24
Member
Author
|
cloud-status: started — baseline 297/0/162 (dev env; 134/0/325 in CI's test-only env); plan: run package doctests in CI (#24), setup-backend skill (#10), LanceDB native hybrid (#17), async registry + native qdrant (#20), README/CLAUDE.md refresh Generated by Claude Code |
LanceDBCollection now satisfies SupportsHybrid. The lexical side runs on LanceDB's native BM25 full-text index over `text` (no tantivy needed), built on the first hybrid call; rows written later are still searched. Fusion stays vd's client-side RRF, so the fused score matches every other backend. Filters apply client-side as on the dense side. Refs #17 (lancedb item; the other five backends remain)
connect_async now dispatches through a registry of native async clients (register_async_backend / list_async_backends) and falls back to the to_thread wrapper otherwise; native=False forces the wrapper. New bases AsyncAbstractCollection / AsyncAbstractClient give native adapters the same user-facing surface as the sync ones. The I/O-free embedding / dimension / query-resolution policy moved from AbstractCollection into a shared _CollectionPolicy mixin so both bases use one implementation (sync behaviour unchanged). qdrant is the first native backend (qdrant_client.AsyncQdrantClient), sharing its filter compiler and point converters with the sync adapter. Tested in embedded :memory: mode, including a parity test against the wrapped sync adapter. hybrid_search_async now works on native collections (native hybrid if present, else async dense + client-side BM25 + RRF). The vd-add-backend dev skill documents the native hybrid and native async hooks, and names the real conftest backend lists. Refs #20 (qdrant done; nine backends remain)
…0.5) pgvector-python 0.5 returns its own non-iterable Vector type from the psycopg adapter, so every document read raised "'Vector' object is not iterable". Convert through to_list()/tolist() when available. Verified against a live Postgres 17 + pgvector server (30 pgvector tests pass, 10 failed before); a server-free unit test guards the conversion. Fixes #26
New user skill vd-setup-backend: the diagnose -> act -> connect -> smoke-test loop, per-archetype start-up (embedded persistence kwargs, Docker one-liners, managed credentials and which adapters read them from the environment), and troubleshooting. Every snippet was run: embedded backends, Milvus Lite, and live Redis 8, Elasticsearch 9 and Postgres 17 + pgvector servers. install_command() now returns `pip install "vd[<backend>]"` for every backend with an adapter, so the printed command installs exactly what the adapter imports (pyproject extras are the single source of truth). The old table had drifted: pgvector omitted psycopg[binary] and milvus omitted milvus-lite. install_backend(run=True) parses the quoted command with shlex. A test checks every adapter has a matching extra. vd-backend-choose now covers choosing only (setup moves to the new skill) and lists native-hybrid and native-async backends. vd-search gains a hybrid-search section and drops a stale claim that chroma returns distances. All skills use spec-clean frontmatter (audience under metadata). Fixes #10
__version__ was a literal "0.2.0" that CI's version bump never touches, so it was stale on every release (0.2.11 today). Read it from importlib.metadata instead. Also fill the empty `authors` field and add a test that every bundled skill's frontmatter is spec-clean. Fixes #27
README now opens with what vd does and an agent section (skills via gh skill or the pip-shipped folder, where CLAUDE.md is, a runnable example), then the essentials, including the new hybrid and async sections, and ends with a contributor section. Every Python example is executed by tests/test_readme.py, which also checks the printed output. CLAUDE.md gains how to run the tests (CI's exact command, the live server suite), the async and hybrid architecture, and a status line on the refactor priorities.
This was referenced Sep 26, 2026
.claude/skills/* symlink into vd/data/skills/. Hatch walks .claude first and skips files whose real path it has already seen, so the sdist kept only the .claude copies and the wheel built from it carried no skills (true of the published 0.2.11). Give the sdist an explicit only-include list so .claude is never walked, and test the sdist file list through hatchling (added to the test extra so CI runs it). Fixes #28
- qdrant: connect_async returns the native client only with url=. In embedded mode qdrant-client's async client runs blocking code inside its coroutines (measured: the loop never got a turn during searches), so embedded mode keeps the thread-pool wrapper. Docs corrected. - lancedb: concurrent first hybrid calls raced creating the FTS index (Lance commit conflict). Creation is now serialized, a conflict is tolerated when the index exists afterwards, the table is reopened so handles opened earlier see the index, and the check is cached. - hybrid_search_async on native collections accepts sync lexical_search callables again (they get a mapping of the documents); async ones get the collection. - hybrid_search warns instead of silently ignoring lexical_search= when the collection runs hybrid natively. - The sdist test normalizes path separators for the Windows job. Refs #17, #20
…ocation) - lancedb: the "FTS index ready" cache is now module-level per (database, table), checked before taking the creation lock, dropped on delete_collection, and a missing-index error on search forgets the entry and rebuilds once. Fixes a stale cache after a table is dropped and recreated, and lock contention for per-request collection objects. - qdrant: an http(s):// location= counts as server mode for connect_async, and gets api_key and extra client kwargs like url= does (the sync adapter dropped them too). - hybrid_search_async documents what sync vs async lexical_search callables receive on native collections. Refs #17, #20
lancedb <= 0.25 raises RuntimeError (not ValueError) when full-text search finds no INVERTED index. Catch both, only for that message, so the one-shot rebuild after a table was dropped behind vd's back works on every supported release (verified on 0.25.0 and 0.39.0). Refs #17
thorwhalen
changed the title
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloud sweep of 2026-09-26. Every change was tested in the cloud VM; see the counts below.
Tests
Command (CI's exact invocation):
python -m pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' --ignore=examples --ignore=scraptestextra only.[test,dev], Python 3.11The baseline never collected the package's own doctests; the final counts include them. The live servers ran in Docker inside the VM, pulled from non-Docker-Hub registries (Docker Hub was rate-limited, so Weaviate, MongoDB Atlas Local and a Qdrant server were not exercised by me; the reviewer ran a temporary Qdrant server).
Dependents, with this branch installed editable over each:
newsmood's 6 failures are identical before and after:
tests/test_clusters.pyraises aconfig2pymissing-config error unrelated to vd.Changes
test:run the package's doctests in CI (testpathsgainsvd; rootconftest.pyskips backend modules whose SDK is missing); fix two stale doctests invd/text.py.feat(lancedb):native hybrid search through LanceDB's built-in FTS index.feat(async):native async backend registry (register_async_backend,list_async_backends,connect_async(native=...)),AsyncAbstractCollection/AsyncAbstractClientbases, native qdrant adapter; shared_CollectionPolicymixin.fix(pgvector):reading documents failed with pgvector-python 0.5.feat:vd-setup-backendskill;install_commandreturns thevd[<backend>]extra; chooser and search skills refreshed; spec-clean skill frontmatter.fix:vd.__version__from package metadata;authorsfilled.docs:AI-first README with every example executed bytests/test_readme.py; CLAUDE.md refreshed.fix(build):the wheel shipped no skills; sdistonly-include.fix:adversarial-review findings, round 1: qdrant's native async client is used only with a server (url=), since the embedded one blocked the event loop; LanceDB first-call index race; synclexical_searchon native async collections; a warning whenlexical_search=is ignored on a native-hybrid path; a Windows path fix in the sdist test.fix:review round 2: LanceDB index cache is module-level per table, invalidated on drop, and rebuilt once on a missing-index error; anhttp(s)://qdrantlocation=counts as server mode and forwardsapi_keyand client kwargs.fix(lancedb):recover from a stale index cache on lancedb 0.25 and older too, which raiseRuntimeErrorthere. Verified on 0.25.0 and 0.39.0.Issues
__version__), Published wheel ships no skills (vd/data/skills is dropped from the sdist) #28 (wheel without skills). CI never runs the package's doctests (and two of them are stale) #24, pgvector backend: reading a document fails with pgvector-python 0.5 ('Vector' object is not iterable) #26, vd.__version__ is hard-coded to 0.2.0 and never updated #27 and Published wheel ships no skills (vd/data/skills is dropped from the sdist) #28 were filed during this sweep with reproductions.needs-locallabel for the account-bound backends.Not done, and why
Adversarial review (Opus subagent)
Three rounds by a separate Opus subagent, briefed to refute the change. It reproduced every suspicion before reporting it. Final verdict: APPROVE. No round had a blocking finding.
Round 1 (APPROVE, 8 non-blocking):
url=or anhttp(s)location=; I measured the loop never getting a turn with the embedded native client. Docs corrected.lexical_searchcallables broke on native async collections. Fixed: they receive a mapping of the documents.lexical_search=was silently ignored on native-hybrid backends, now including LanceDB. Fixed: it warns.AbstractCollection.supported_filter_operatorson the base class itself, removedQdrantCollection._payload,._pointand._to_document. Accepted: private, and no dependent uses them.Round 2 (APPROVE): the index cache went stale after a drop and recreate, and the lock was contended for per-request collection objects. Fixed in round 2's commit. The warning's stack location on async paths is accepted as is.
Round 3 (APPROVE): older LanceDB raises
RuntimeErrorfor the missing index. Fixed in the last commit, a narrow follow-up verified on both versions. A table dropped by another thread mid-search surfaces LanceDB's own not-found error. Accepted: that is concurrent misuse, not caused by this PR.For whoever lands it
install_command()output changed from raw client names topip install "vd[<backend>]". No dependent parses it.connect_async("qdrant", url=...)now returns a native client (native_async is True, no.syncattribute);native=Falsegives the old wrapper. Embedded qdrant is unchanged.testextra, so backend-specific tests (qdrant, LanceDB, pgvector, live servers) skip there. They were run in this VM. Consider a CI extra withqdrant-clientandlancedb, both embedded and server-free.Fixes #24
Fixes #10
Fixes #26
Fixes #27
Fixes #28
🤖 Generated with Claude Code