Problem
After the recent refactoring of the heading hierarchy detection pipeline (moving from reason-code mapping to per-chunk independent LLM calls), several inconsistencies and stale code paths have been identified that affect both the quality of heading detection and the reliability of the LLM-failure fallback path.
User-Facing Symptoms
-
Inconsistent debug output: The heading detection debug CSVs show NEG condition arrays of varying widths across different rows of the same document, making it difficult for document processing engineers to diagnose heading misclassification issues.
-
Over-aggressive heading filtering: Certain heading candidates (e.g. numbered headings ending with punctuation like 1.2 System Design.) are being prematurely filtered out during the initial scan rather than being sent to the LLM for proper evaluation. This can result in missing sections in the final parsed document structure, especially for PPTX-to-MD converted documents and DOCX files without explicit heading styles.
-
Degraded fallback quality: When the LLM hierarchy detection service is unavailable or times out, the fallback produces malformed hierarchy trees with invalid level values (-2), leading to broken document navigation and section structures.
Root Cause
The refactoring consolidated multiple NEG conditions (including end-of-line punctuation checking) into an always-on mode, whereas the original design intentionally deferred punctuation checking to a second-pass judge_negs stage to avoid false negatives during the initial candidate selection. Additionally, hardcoded condition counts fell out of sync with the actual condition arrays, and dead code from the removed collapse pipeline stage was left behind.
Expected Behavior
- NEG condition arrays should have consistent width across all rows
- End-of-line punctuation should only be checked during the second-pass
judge_negs to minimize false filtering of legitimate heading candidates
- The LLM-failure fallback path should produce valid hierarchy levels suitable for tree construction
- Stale code paths from removed pipeline stages should be cleaned up
Problem
After the recent refactoring of the heading hierarchy detection pipeline (moving from reason-code mapping to per-chunk independent LLM calls), several inconsistencies and stale code paths have been identified that affect both the quality of heading detection and the reliability of the LLM-failure fallback path.
User-Facing Symptoms
Inconsistent debug output: The heading detection debug CSVs show NEG condition arrays of varying widths across different rows of the same document, making it difficult for document processing engineers to diagnose heading misclassification issues.
Over-aggressive heading filtering: Certain heading candidates (e.g. numbered headings ending with punctuation like
1.2 System Design.) are being prematurely filtered out during the initial scan rather than being sent to the LLM for proper evaluation. This can result in missing sections in the final parsed document structure, especially for PPTX-to-MD converted documents and DOCX files without explicit heading styles.Degraded fallback quality: When the LLM hierarchy detection service is unavailable or times out, the fallback produces malformed hierarchy trees with invalid level values (-2), leading to broken document navigation and section structures.
Root Cause
The refactoring consolidated multiple NEG conditions (including end-of-line punctuation checking) into an always-on mode, whereas the original design intentionally deferred punctuation checking to a second-pass
judge_negsstage to avoid false negatives during the initial candidate selection. Additionally, hardcoded condition counts fell out of sync with the actual condition arrays, and dead code from the removedcollapsepipeline stage was left behind.Expected Behavior
judge_negsto minimize false filtering of legitimate heading candidates