Add end-to-end pipeline tests; fix compaction in manage.py#8
Merged
Conversation
New tests (28 → 55 cases, coverage 39% → 55%), all local, no MinIO: - tests/image/test_pipeline_e2e.py: real Stage 1→2→3→4→5 chain on a temp dir — synthetic face/blurred/no-face images plus corrupt bytes and a missing path; asserts statuses, verdicts (null for failures), blob v2 storage, time index, queries, and delete. Skipped when the SCRFD model pack is absent. - tests/workflow/test_index_manage.py: ZONEMAP + IVF_FLAT index creation, missing-column error, delete_by_date bounds/window. - tests/storage/, tests/audio/test_prompt.py: read_manifest formats, lance_storage_options, validate_blob_v2 failure path, prompt builder. The e2e tests exposed two real defects in workflow/manage.py, fixed here: - lance_ray 0.4.x compact_files crashes unless compaction_options is an explicit dict (its None default is rejected downstream). - Compaction of blob v2 tables fails inside lance's decoder with current versions regardless of engine; compaction is now best-effort with a warning so deletion still succeeds. Documented in README.
added 2 commits
July 6, 2026 16:02
uv-based setup with dependency caching; SCRFD-dependent tests skip automatically when the model pack is absent, so CI needs no model download.
Review found test_build_embedding_index_small_table failing in environments with a pre-existing Ray cluster: Ray's default init joins it, and the foreign workers cannot deserialize this venv's lance_ray functions. Two-layer fix: - local_ray fixture forces a fresh local cluster from this venv (clears RAY_ADDRESS, shuts down any existing context). - The test is marked `ray` and excluded from the default run via pyproject addopts; run explicitly with `pytest -m ray`. The delete_by_date tests stay in the default suite — compaction is best-effort, so they pass even where Ray is unusable. The CI workflow (cherry-picked from PR #9) runs the default suite as a blocking step and `-m ray` as a non-blocking one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds automated end-to-end regression for the pipeline (previously only verified manually against MinIO), plus quick-win unit tests. All tests run fully local — no MinIO required (
daft.functions.downloadhandles local paths, failures → null).28 → 55 test cases, line coverage 39% → 55%.
New tests
tests/image/test_pipeline_e2e.py— the real Stage 1→2→3→4→5 chain in a temp dir: synthetic sharp/blurred portraits + no-face image + corrupt bytes + missing manifest path. Asserts one row per manifest entry, status breakdown (ok×3 /decode_failed/download_failed), verdicts (null for failed rows, not false), blob v2 validation, blob presence matching status, ZONEMAP index, scalar/SQL queries on the real pipeline output, and date-range deletion. Skipped when the SCRFD model pack is absent (CI-safe).tests/workflow/test_index_manage.py— media-agnostic stages on a 300-row local table: IVF_FLAT index (small-table params), missing-embedding-column error, ZONEMAP,delete_by_datebefore/after/window/no-bound.tests/storage/,tests/audio/test_prompt.py—read_manifest(parquet/jsonl/csv/unsupported),lance_storage_options,validate_blob_v2failure path, prompt builder enums.Bugs found and fixed (workflow/manage.py)
The e2e tests immediately exposed that Stage 5 compaction had never actually worked:
lance_ray.compact_filescrashes unlesscompaction_optionsis an explicit dict — lance_ray 0.4.x passes itsNonedefault straight intoCompaction.plan(), which rejects it.[warn]so deletion still succeeds; documented in README's known limitations.Coverage notes
Pipeline glue went from 0–39% to:
image/workflow/analyze81%,image/workflow/ingest80%,workflow/manage78%,workflow/index54%,storage/io93%,storage/blob100%,audio/prompt100%.image/udfs.pystill reports 26% because@daft.clsUDF bodies execute in Daft worker processes that coverage doesn't instrument — the e2e chain does execute them. Audio ASR/embedding (model-download-dependent) remain out of scope, to be covered when the audio side next changes.