Skip to content

refactor: decompose process_pdf_statement into pure extraction + persistence #198

Description

@boscorat

Parent issue

#194 — function purification

Problem

process_pdf_statement (statements.py:828) is a 9-parameter monolithic function that orchestrates:

  • PDF parsing (creates Statement)
  • 4 temp parquet file writes (BatchLines, StatementHeads, StatementLines, ChecksAndBalances)
  • Error classification
  • Timing measurement
  • Prints to stdout/stderr on failure

The function mixes pure data construction with side-effecting I/O, making it hard to test error paths in isolation.

Proposal

Split into (a) pure extraction: extract_statement_data(pdf, config) -> ExtractionResult, (b) pure classification: classify_result(extraction) -> PdfResult, (c) side-effecting persistence: write_temp_files(result, paths). Timing and prints should be in the orchestration layer. Also convert the batch_line dict to a typed dataclass.

Code relationships

Target function:

  • process_pdf_statement()statements.py:828

Callers (will need updating):

  • StatementBatch.__process_batch_sequential()statements.py:1377
  • StatementBatch.__process_batch_turbo()statements.py:1398 (via ProcessPoolExecutor)

Related (called from process_pdf_statement):

  • Statement.__init__()statements.py:347
  • _handle_parquet_write_error()statements.py:800
  • BatchLines, StatementHeads, StatementLines, ChecksAndBalancesparquet.py

Existing tests

  • tests/test_statements.py::TestGoodStatements — exercises process_pdf_statement indirectly via StatementBatch
  • tests/test_statements.py::TestBadStatements — exercises error paths indirectly

No direct unit tests exist for process_pdf_statement.

Siblings

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    refactorcode refactoring

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions