Resolve Issue #158: Page-Memory PR4 modules, Excel table pathing, evidence renderer optimization - #159
Merged
Conversation
… ProfileAgent. - Added "page_locate" to BudgetStage for improved budget management. - Introduced lazy loading for ProfileAgent to optimize imports in document_agent. - Updated tools initialization to include page_locate functionality.
New page_memory modules: - C1 page_renderer: PNG + thumbnail + raw_text per page - C2 page_plan: rule-based vlm_lite/text_only/skip_tagging strategy - C3 page_tagger: VLM per-page annotation with JSON retry + blurry degradation - C6 page_section_mapper: skeleton × tagger → section_path (primary/spans/inherited) - C7 memory_service: unified page/shard_page builder via full C1-C7 pipeline shared-python GAP fixes: - GAP-1: zip_chunk_schema recognizes 'page' chunk type (no collapse to text) - GAP-2: zip_result_resources collects pages/ directory - GAP-3: zip_doc_navigation counts page_chunks in stats
Field changes: - content = raw PyMuPDF text only (no [SUMMARY]/[RAW] markers) - summary = VLM or LLM-generated summary (metadata only) - keywords = VLM or summary-full LLM extracted (semicolon-separated) - kind = PageLabel.kind from Profile Agent (not plan.reason) - observed_titles = from C4 skeleton primary titles (not VLM) - Remove thumb_uri (only page_image_uri kept) - Remove status field (strategy_used covers quality info) Strategy changes: - text_only: calls existing summary-full LLM for summary+keywords - skip_tagging: preserves raw text content, marks EMPTY if blank - vlm_lite: outputs summary+keywords (no observed_titles) - Mapper no longer depends on PageTagResult
… support, and optimized evidence rendering with configurable character limits
…ptional ctx/last_verify narrowing)
EricNGOntos
added
the
page-memory-pipeline
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| actual_max_depth = max_title_depth(nodes) | ||
| emit_depth = min(actual_max_depth, self.config.max_emit_depth) | ||
| emit_depth = max(emit_depth, self.config.min_emit_depth) | ||
| selected_nodes = nodes |
| subtable_index = int(attrs.get("subtable_index") or 1) | ||
| except (TypeError, ValueError): | ||
| subtable_count = 1 | ||
| subtable_index = 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #158
Changes Overview
This PR delivers four independent but cohesive improvements across the document parsing and retrieval pipeline. All page-memory changes are gated behind
RETRIEVAL_PAGE_MEMORY_ENABLED=false(default) andparse_track='chunk'(default), ensuring zero impact on existing production parsing.1. Page-Memory Per-Page Parsing Pipeline (C1–C7)
New package
apps/worker/app/services/page_memory/with complete per-page processing:page_renderer.pypage_plan.pypage_tagger.pyskeleton_extractor.pypage_section_mapper.pymemory_service.pyGating:
RETRIEVAL_PAGE_MEMORY_ENABLED=falsein API → 400 rejection.documents.parse_trackdefaults tochunk→ Worker never enters page path.2. Document Agent: Page Locate Stage + Lazy Loading
hierarchy_locator.py: Multi-strategy heading-to-page resolver (strict anchoring → grep → VLM verify)page_locate_agent.py/page_locate_subagent.py: Bounded ReAct loop for residual headings that strict anchoring cannot resolvepage_locate_tools.py: grep.title_pages and verify.section_page tool implementationscoordinator.py: Addedpage_locatebudget stage envelope (min=0, cap=0 — inactive by default)__init__.py:ProfileAgentconverted to lazy import via__getattr__to avoid heavy dependency loading3. Excel Table Parsing Enhancement
filename/sheet_name/subtable_titleinstead of flattables/table-name.htmlsheet_name,subtable_index,subtable_count,subtable_titlecontentfield (was reference string)table_section_candidates4. Agentic Evidence Renderer Optimization
RETRIEVAL_AGENTIC_INLINE_TABLE_CHAR_LIMIT(default 10,000 chars) are replaced with a compact summary (asset URL, path, summary, keywords)RETRIEVAL_AGENTIC_INLINE_TABLE_CHAR_LIMITinAIConfig(env-configurable)5. Supporting Changes
zip_chunk_schema.py: Addedpagechunk type counting and formattingzip_doc_navigation.py: Page chunks participate in section tree; table fallback for text-empty documentszip_result_resources.py: New utility for page result resourcesTesting
make checkpasses: 0 lint errors, 0 type errorstest_agentic_evidence_renderer_contract.py,test_excel_parser_contract.pyRisk Assessment