FEAT: Parallel PDF Processing and Extraction Pipeline Reliability Improvements #65

The current extraction pipeline has several throughput and reliability gaps that block running larger batches against the partner's corpus:

• PDFs are processed strictly one at a time, leaving multi-core hardware idle during long batch runs.
• Noisy artifacts in the cleaned text — DOIs, figure captions, numbered references — are being fed to the LLM and diluting the relevant signal.
• Section headers are getting stripped during text cleaning, costing the model useful structural context.
• Weak or low-signal paragraphs are passed to the LLM unfiltered, increasing context usage without improving extraction quality.
• The LLM occasionally returns null fields with no retry, so a single transient failure drops data for that paper.
• Long papers exceed the current Ollama context limit and get silently truncated.

Proposed Changes

• Add a --workers N flag for parallel PDF processing (default 1 to preserve current behavior).
• Strip DOIs, figure captions, and numbered references during cleaning; preserve section headers.
• Add classifier-based paragraph scoring and drop paragraphs with no positive or negative signal.
• Switch the extraction model to qwen2.5:7b and raise the Ollama context limit.
• Add retry logic for null LLM responses.
• Load SpellChecker() once at startup rather than on every call.
• Enable GPU for XGBoost training when available.
• Add an OCR-bypass option for parallel runs where OCR was freezing workers.

Acceptance Criteria

• Pipeline accepts --workers N and runs N files in parallel without breaking single-PDF behavior.
• Cleaned text no longer contains DOI strings, figure captions, or numbered reference entries.
• Section headers survive the cleaning pass.
• LLM null returns trigger at least one retry before the field is recorded as null.
• A long-document test case passes end-to-end without context-window truncation.