feat: TOC link attach, 1-based pages, and profile stage wiring - #291
Merged
Merged
Conversation
- Introduced `_clear_toc_anchor_state` method to reset TOC anchor-related attributes in `ProfileCoordinator`. - Updated `PageAnatomyMap` and `AgentBlackboard` to include new fields for `skeleton_anchor`, `skeleton_nodes`, and `pending_skeleton_anchors`. - Integrated `run_toc_anchoring` in the extraction pipeline to improve TOC processing. - Added serialization and deserialization functions for `skeleton_anchor` and `title_node` to support new data structures. - Refactored `extract_section_skeletons` to utilize persisted `skeleton_anchor` for section boundary resolution.
- Added `rapidocr-onnxruntime` as a dependency for OCR capabilities. - Implemented `_run_text_scan` method in `ProfileCoordinator` to handle text extraction from PDF pages. - Updated `PageAnatomyMap` to include `page_full_text_cache` for storing scanned text. - Enhanced TOC anchoring logic to utilize cached page text for improved accuracy. - Refactored various tools to leverage the new text scanning functionality, ensuring seamless integration with existing workflows.
- Removed `run_toc` and `_run_toc` methods from `ProfileCoordinator` to streamline TOC processing. - Eliminated `page_full_text_cache` from `PageAnatomyMap` and related logic to simplify data handling. - Updated tests to reflect changes in TOC extraction and anatomy map structure, ensuring consistency in functionality.
- Removed `min_pages` parameter from `validate_shard_plan` and related functions to simplify validation logic. - Updated `ProfileCoordinator` to clarify comments regarding single-shard placeholder usage. - Refactored `propose_shard_plan` and `validate_anatomy_map` to enhance clarity and maintainability. - Adjusted tests to reflect the removal of `min_pages` and ensure consistency with updated validation logic.
- Introduced `_attach_toc_page_links` method to attach hyperlinks to TOC entries before calibration. - Updated `run_toc_anchoring` to conditionally skip anchoring based on settings. - Refactored `ProfileCoordinator` to ensure asset probes are executed in the correct order relative to TOC processing. - Enhanced `collect_toc_page_links` to normalize page link destinations and improve link matching logic. - Updated `profile_document` to include a new parameter for skipping TOC anchoring during profiling. - Adjusted tests to validate the new TOC processing flow and ensure correct asset probing order.
- Moved import statements for `anchor_hierarchy`, `finalize_calibration_result`, `pick_primary_offset`, `calibrate_offset`, and `graft_contained_toc` to the top of the file for better readability. - Removed redundant import statements to enhance clarity and maintainability of the `toc_anchoring.py` file.
Lazy imports in toc_anchoring broke module-level mocks; point patches at orchestrator/service/procedure instead. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
|
|
||
| def test_classify_toc_relationship_is_not_on_c4_module() -> None: | ||
| import app.services.page_memory.skeleton_extractor as skeleton_extractor |
Contract tests were still patching symbols on toc_anchoring that are only imported inside functions after the circular-import refactor. Co-authored-by: Cursor <cursoragent@cursor.com>
api and worker both use top-level `app`, so a combined pytest run can leave stale module bindings; patch the defining function globals and run the suites separately in CI. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_links()dest pages as already 1-based (no off-by-one+1).stop_after_asset_probe,skip_toc_anchoring) and run asset probe before TOC.Test plan
make checkpytestTOC graft + link attach/match contractsMade with Cursor