Improve ingestion pipeline

[shredEngineer]

We could streamline the ingestion pipeline as implemented through archive_agent/data/FileData and the /archive_agent/data/loaders subpackage.

Options:

• Markitdown
• Marker

Goals:

• Support more file types
• Improve ingestion quality, e.g. retain PDF hierarchy (could also tweak currently used prompt)

---

For Markitdown there are some points to consider:

• PDF extraction seems to use pdfminer instead of pymupdf, which has been reported to be slower.
• The image description feature seems very basic compared to combined OCR and entity extraction, so Archive Agent's native method as currently implemented seems to be superior.

For marker, I have to do similar research.

---

ESSENTIAL:

Make sure the new module is thread-safe.

The currently used pymupdf module is not thread safe, which is a performance hit.
PDFs are currently handled differently in IngestionManager due to this.

---

Also check out MinerU: https://github.com/opendatalab/MinerU

[shredEngineer]

Base Libraries

These are lower-level libraries focused on direct extraction of text and images from PDFs without advanced formatting like markdown output or built-in OCR for image-to-text conversion.

Library	Speed	Thread Safety	Text Extraction	Image Extraction	Notes
pypdf (formerly PyPDF2)	Medium (15x slower than PyMuPDF)	Yes (pure Python)	Yes	Yes	Reliable for basic tasks; improved recently for accuracy; no C dependencies.
pdfplumber	Slow (similar to pdfminer)	Yes (pure Python)	Yes (strong for tables)	Yes	Built on pdfminer; good for layout-aware extraction.
pdfminer.six	Slow (35x slower than PyMuPDF)	Yes (pure Python)	Yes	Yes	Detailed layout analysis; handles complex structures well but verbose.
pypdfium2	Very Fast (comparable to PyMuPDF)	Likely yes (PDFium designed for multi-threaded environments like Chrome)	Yes	Yes (via rendering)	High-quality text extraction; liberal license; binding to PDFium.
PyMuPDF	Very Fast	No (but supports multiprocessing)	Yes	Yes	Excellent for speed and versatility; your current choice, but avoid shared use in threads to prevent crashes.

Higher-Level Libraries

These build on base libraries and often output structured formats like markdown. Many include OCR for converting images to text, making them suitable for scanned PDFs.

Library	Speed	Thread Safety	Text Extraction	Image Extraction	Notes
Marker	Fast (optimized for batch processing)	Likely yes (supports parallel file conversion)	Yes (to markdown/JSON)	Yes (with OCR via Tesseract)	ML-based layout detection; handles books/scientific papers; converts images to text descriptions.
Docling	Fast	Likely yes	Yes (to markdown)	Yes (with OCR support)	IBM open-source; efficient for RAG pipelines; handles scanned docs well.
Unstructured	Variable (slower with OCR)	Yes (supports parallel partitioning)	Yes (to elements/markdown)	Yes (with OCR)	Great for RAG/LLM prep; partitions into text/tables/images; flexible but resource-intensive.
PyMuPDF4LLM	Very Fast (inherits from PyMuPDF)	No (same as PyMuPDF)	Yes (to markdown)	Limited (no built-in OCR)	Lightweight extension of PyMuPDF; detects tables/text order; no image-to-text.
OCRmyPDF	Medium (depends on Tesseract)	Yes (multithreaded with --jobs)	Yes (adds OCR layer)	No direct (focuses on text overlay)	Outputs searchable PDFs; integrates Tesseract for scanned docs; not native markdown.
MarkItDown	Unknown (recent release)	Likely yes	Yes (to markdown)	Yes	Microsoft package; handles various formats; limited benchmarks available.
MagicConvert	Unknown	Likely yes	Yes (to markdown)	Yes (with OCR)	Supports multiple doc types; OCR via external tools; less documented performance.

[shredEngineer]

Also introduce a minimum chunk length, i.e. join adjacent chunks when it's too short