Skip to content

Parser document state still keeps only deduped chunks #31

Description

@suguanYang

Problem

Trace 019e1abc4a46b94ec4874f74d5ec13ac for staging job job_f2fcb52b2db7 showed the parser produced and exported 72 chunks, but final document/retrieval state only published 11 chunks because publication dedup reused the deduped chunk list for document_chunks.

Expected

Result ZIP, job_chunks, and document_chunks should all preserve the full parsed result chunk set. Deduplication should remove duplicate local media after export and record stats, but it should not shrink document-facing chunk state.

Evidence

Fix Direction

Keep publication dedup stats and local media cleanup, but publish the full normalized parsed chunk list during job finalization.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions