Deliverable 3 pilot findings: 234 issues across 30 documents — prioritized improvement plan #82

Activity

1. dylan-isaac commented on Mar 23, 2026

   

   dylan-isaac

   on Mar 23, 2026

   CollaboratorAuthor

   More actions

   Update: Hyperlink Extraction — Docling Status & pypdfium2 Fallback Plan

   Docling Status (as of March 2026)

   The DoclingDocument schema does have a TextItem.hyperlink: Optional[Union[AnyUrl, Path]] field (docling-core source), and the low-level parser (docling-parse v5) extracts PDF link annotations. However, the propagation step — matching hyperlink bounding boxes to text items — was never implemented for the PDF pipeline.

   • Issue #828 (open since Jan 2025): Original report. Maintainer confirmed hyperlinks are identified in docling-parse but need propagation to DoclingDocument.
   • PR #3131: External contributor added spatial matching of hyperlink annotations to text clusters. CI passes, one maintainer approved, but it's stuck awaiting a second reviewer and is not merged.

   Result: Every TextItem in our Docling JSON response has hyperlink=null for PDF documents. URLs that appear as literal text in the PDF body are preserved, but clickable hyperlinks where display text (e.g., "click here") links to a URL are lost entirely.

   pypdfium2 Fallback Implementation Plan

   Since we already use pypdfium2.raw in pdf_classifier.py for annotation inspection (form field widget counting), we can use the same API pattern to extract hyperlinks. Here's the plan:

   Step 1: Extract link annotations from PDF bytes

   Add a new function extract_hyperlinks() to a new module (e.g., src/services/hyperlink_extractor.py) or extend pdf_classifier.py:

   import pypdfium2
   import pypdfium2.raw as pdfium_c
   import ctypes
   
   @dataclass
   class PdfHyperlink:
       page_number: int       # 1-indexed
       url: str               # Target URI
       bbox: tuple[float, float, float, float]  # (left, bottom, right, top) in PDF coords
   
   def extract_hyperlinks(file_content: bytes) -> list[PdfHyperlink]:
       """Extract hyperlink annotations from PDF using pypdfium2."""
       pdf = pypdfium2.PdfDocument(file_content)
       links: list[PdfHyperlink] = []
       
       for page_idx in range(len(pdf)):
           page = pdf.get_page(page_idx)
           try:
               # Enumerate link annotations on the page
               start_pos = ctypes.c_int(0)
               link_ptr = ctypes.c_void_p()
               
               while pdfium_c.FPDFLink_Enumerate(page.raw, ctypes.byref(start_pos), ctypes.byref(link_ptr)):
                   # Get link bounding rectangle
                   rect = pdfium_c.FS_RECTF()
                   if not pdfium_c.FPDFLink_GetAnnotRect(link_ptr, ctypes.byref(rect)):
                       continue
                   
                   # Get action and check if it's a URI action (type 1)
                   action = pdfium_c.FPDFLink_GetAction(link_ptr)
                   if not action:
                       continue
                   if pdfium_c.FPDFAction_GetType(action) != 1:  # PDFACTION_URI
                       continue
                   
                   # Extract the URI string
                   buf_size = pdfium_c.FPDFAction_GetURIPath(pdf.raw, action, None, 0)
                   if buf_size <= 0:
                       continue
                   buf = ctypes.create_string_buffer(buf_size)
                   pdfium_c.FPDFAction_GetURIPath(pdf.raw, action, buf, buf_size)
                   url = buf.value.decode("utf-8", errors="replace")
                   
                   links.append(PdfHyperlink(
                       page_number=page_idx + 1,
                       url=url,
                       bbox=(rect.left, rect.bottom, rect.right, rect.top),
                   ))
           finally:
               page.close()
       
       pdf.close()
       return links

   This follows the exact same pattern as _extract_metadata() in pdf_classifier.py (lines 278-302), which already uses pdfium_c.FPDFPage_GetAnnotCount, FPDFPage_GetAnnot, and FPDFAnnot_GetSubtype.

   Step 2: Spatial-join hyperlinks to Docling text

   The Docling JSON response includes bounding boxes for text items (body → items → prov → bbox). We can match hyperlink rectangles to the text they cover:

   • For each PdfHyperlink, find text items on the same page whose bounding box overlaps with the link rectangle
   • The overlapping text becomes the display text for the markdown link
   • Inject [display text](url) into the per-page markdown at the corresponding position

   This spatial join would happen in _step_docling() (pipeline_viewer.py ~line 700), right after extracting figures and before the pipeline proceeds to AI processing steps.

   Step 3: Inject into per-page markdown

   After identifying which text spans are linked, modify the per-page markdown (v0) to wrap linked text in markdown link syntax. This runs as a deterministic post-processing step — no LLM needed.

   Step 4: Future-proof for Docling native support

   Structure the code so that when PR #3131 merges and TextItem.hyperlink becomes populated:

   1. Check json_content for populated hyperlink fields first
   2. Fall back to pypdfium2 extraction only if Docling doesn't provide them
   3. Eventually remove the pypdfium2 fallback entirely

   def get_hyperlinks(file_content: bytes, json_content: dict) -> list[PdfHyperlink]:
       """Get hyperlinks from Docling JSON if available, else fall back to pypdfium2."""
       docling_links = extract_hyperlinks_from_docling_json(json_content)
       if docling_links:
           return docling_links
       return extract_hyperlinks_from_pypdfium2(file_content)

   Where this fits in the pipeline

   _step_docling():
       1. Docling extraction + pypdfium2 page rendering (parallel) — existing
       2. Split markdown by page — existing
       3. Extract figures from JSON + crop — existing  
       4. Replace image placeholders — existing
       5. ✨ NEW: Extract hyperlinks (pypdfium2) + spatial-join + inject into markdown
       6. Detect column layout — existing
   

   The hyperlink injection happens once, deterministically, at the end of _step_docling(). All subsequent pipeline steps (structure analysis, page correction, boundary fixes, cleanup) will see the hyperlinks already in the markdown and preserve them.
2. dylan-isaac

   added a commit that references this issue on Apr 22, 2026

   docs(benchmarks): publish Deliverable 3 pilot benchmark (#126)

   ...

   def0537