feat(doc-agent): Modular Document Anatomy Agent — bootstrap, planner, executor, persist - #108
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| from app.services.document_agent.planner import ProfilePlanner | ||
| from app.services.document_agent.registry import REGISTRY | ||
| from app.services.document_agent.state import AgentBlackboard, DocumentAgentState | ||
| from app.services.document_agent import tools as _registered_tools # noqa: F401 |
| finally: | ||
| try: | ||
| doc.close() | ||
| except Exception: |
| finally: | ||
| try: | ||
| doc.close() | ||
| except Exception: |
| finally: | ||
| try: | ||
| doc.close() | ||
| except Exception: |
| finally: | ||
| try: | ||
| doc.close() | ||
| except Exception: |
| logger.debug(f"parse agent trace flush failed: {exc}") | ||
| try: | ||
| self._db.rollback() | ||
| except Exception: |
| finally: | ||
| try: | ||
| doc.close() | ||
| except Exception: |
| if isinstance(item, dict) and "id" in item and "level" in item: | ||
| try: | ||
| llm_levels[int(item["id"])] = item["level"] | ||
| except (TypeError, ValueError): |
Introduce Phase-0 anatomy infrastructure for large-PDF splitting: - page_map.py: data contracts — PageMap, PageFeature, Shard, CutPoint, H1BoundaryResult, H1Match - agent.py: DocumentAnatomyAgent — LLM tool-calling loop (scan → find_h1 → propose_cuts → finalize) with deterministic fallback - tools/scan_all_page_features.py: full-page PyMuPDF structural feature extraction (text density, image coverage, table count, orientation) in an isolated child process - tools/find_h1_boundaries.py: TOC-page grep + body grep to locate level-1 heading physical pages; falls back to preview grep when no TOC is detected - shard_splitter.py: split_pdf_by_shards() and merge_shard_dataframes() with page_nums offset correction Integration into parse_pdfs() is the next step.
Delete unused Phase 1 modules that were never integrated into the production parser pipeline: - manifest.py: ShardManifest / ShardSignal / GlobalSignals / SpecialPage - tools/propose_shard_plan.py: LLM + fallback shard planning - tools/probe_sample_pages.py: stratified page sampling - tools/probe_vlm_inspect.py: VLM page screenshot inspection Adapt surviving modules: - classify_special_pages.py: inline SpecialKind Literal type (was imported from deleted manifest.py) - __init__.py / tools/__init__.py: remove all Phase 1 exports - page_map.py / scan_all_page_features.py: clean stale docstring refs The Phase 0 DocumentAnatomyAgent (agent.py, page_map.py, scan_all_page_features, find_h1_boundaries, shard_splitter) is the sole path forward. No production impact: neither Phase 0 nor Phase 1 was ever called from parse_pdfs() or any other production entry point.
This commit removes the following unused files and classes related to the DocumentAnatomyAgent: - agent.py: Deleted the DocumentAnatomyAgent class, which was not integrated into the production pipeline. - page_map.py: Removed PageMap, PageFeature, and related classes that were part of the legacy structure. - tools: Deleted all tools related to page feature extraction and heading boundary detection, including scan_all_page_features.py and find_h1_boundaries.py. Updated the __init__.py files to reflect these changes and cleaned up imports accordingly. This refactor simplifies the codebase and focuses on the current implementation of the DocumentAnatomyAgent.
…ine transitions This commit modifies the DocumentAgent's state management and tool integration by replacing the legacy H1 boundary and TOC page detection tools with a new boundary candidate system. Key changes include: - Updated the state transitions to utilize `collect.boundary_candidates` instead of the removed `find.h1_boundaries` and `find.toc_pages`. - Introduced a new `BoundaryCandidate` data structure in the manifest to encapsulate boundary candidate details. - Adjusted the `ProfileCoordinator` and validation logic to accommodate the new boundary candidates. - Enhanced the `ParseRunRecorder` to include detailed trace information for boundary candidates. - Removed obsolete tools related to H1 and TOC detection, simplifying the toolset. These changes aim to improve the clarity and efficiency of the DocumentAgent's processing pipeline.
…es into document agent state.
…, planner, and persist components
…rd planning resilience with validation fallback logic.
…nd implement PDF shard splitting and merging logic.
…ion logic for markdown parsing
…n up stale pipeline code - Remove hardcoded num_pos/num_neg params; derive zero-filled arrays dynamically - Remove dead 'Sure' return branch from get_max_lvl, add -> int type annotation - Restore include_punc=False default in remove_by_conditions to defer punc checking to judge_negs second pass (reduces false heading filtering) - Remove dead 'collapse' task code and rename functions for clarity - Simplify redundant regex character class in NEG rule 3 - Normalize -2 -> 1 in est_hierarchies_naive for valid LLM-failure fallback Closes #112
EricNGOntos
force-pushed
the
feat/wuchengke/dev
branch
from
May 28, 2026 04:31
2c7f6f1 to
f34958a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a comprehensive modular restructure of the Document Anatomy Agent, the pre-parse intelligence layer that profiles PDF documents before they enter the main parsing pipeline.
Key Changes
Architecture Refactoring (5 commits + 1 type-fix)
agent.pyinto four clean sub-packages:bootstrap/— page feature aggregation, classification, and initial probingplanner/— LLM-driven planning with DAG-style step decompositionexecutor/— ReAct-loop tool execution enginepersist/— anatomy map persistence layerprobe_sample_pages,probe_vlm_inspect,scan_all_page_features,classify_special_pages,find_h1_boundaries) with new modular tools:find_toc_anchor_pages— locate TOC pages via text heuristicsvlm_toc_extractor— VLM-based TOC extraction with structured outputextract_toc_with_boundaries— merge TOC entries with boundary candidatesmatch_h1_pages— H1-to-page matching using text extractioninspect_pages— generic VLM inspection with budget controlgrep_text— text search across extracted page textclassify_page_kinds— rule-based page type classificationprobe_page_features— per-page feature extraction (font clusters, image counts)validate_anatomy_map/persist_anatomy_map— validation and persistenceverdict— final agent decision toolstate.pyblackboard pattern with typedBlackboarddataclassbudget.py) with visual/text/planning poolstrace.py) for debugging and analyticscoordinator.pyorchestrates the multi-phase agent loopShared Layer Changes (Additive Only)
openai_compatible_client_sync.py: Addedchat_completion_raw_with_usage()method that returns the raw OpenAI response object alongside usage stats. The existingchat_completion_with_usage()is preserved unchanged — internally refactored to delegate to a shared_make_ali_pool_raw_call()helper.ParseRun,ParseStep,DocumentPagePlan— dedicated tables for doc-agent trace persistence. No modifications to any existing tables.f8a9b0c1d2e3_add_parse_agent_tables.pycreates 3 new tables only.Cleanup
shard_splitter.pyfromdocument_parser/formats/pdf/(zero external references)page_map.py(replaced by new state model)apps/worker/scripts/to.gitignorefor local debug scriptstest-doc-agentMakefile targetProduction Safety
This PR does NOT touch the main parsing production pipeline:
parse_service.py, any format parser,pred_titles(), or retrieval engine_make_ali_pool_raw_call) preserves identical behavior for all existing callersQuality Checks
ruff check: ✅ All checks passed (tracked files)pyright: ✅ 0 errors, 0 warnings