Skip to content

fix: preserve full parsed chunks in result ZIP - #30

Merged
suguanYang merged 1 commit into
stagingfrom
fix/wangbinqi/preserve-result-zip-chunks
May 12, 2026
Merged

suguanYang merged 1 commit into
stagingfrom
fix/wangbinqi/preserve-result-zip-chunks

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • keep the full parsed chunk list for result ZIP generation, raw artifact upload, and job_chunks
  • compute a separate deduped publication chunk list for retrieval document_chunks and graph publication
  • move local media GC until after the result ZIP and raw artifacts are uploaded so duplicate image/table assets are not removed before export
  • add a worker contract regression covering DB dedup with duplicate text and image chunks

Closes #28

Tests

  • uv run pytest apps/worker/tests/contract/test_parse_task_contract.py::test_should_export_full_result_when_publication_deduplicates_existing_chunks
  • uv run pytest apps/worker/tests/contract/test_parse_task_contract.py
  • uv run --all-packages --group lint ruff check apps/worker/app/core/tasks/kb_tasks.py packages/shared-python/shared/services/job_lifecycle_sync.py apps/worker/tests/contract/test_parse_task_contract.py
  • uv run --all-packages --group typecheck pyright --project pyproject.toml apps/worker/app packages/shared-python/shared

@suguanYang
suguanYang merged commit 9af5c8d into staging May 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant