Task: Refactor Code Duplication and Enforce Similarities Prevention

[nlebovits]

Describe the task

Use pylint's similarities detection to identify and refactor duplicated code throughout the data/src/ pipeline, then update the pre-commit configuration to prevent future code duplication. The current pre-commit hook for similarities checking is non-blocking and reveals multiple opportunities to reduce code duplication and technical debt. This task involves running the pylint similarities analysis, refactoring identified duplications, and then switching the pre-commit hook from non-blocking to blocking mode to enforce code quality standards and prevent new duplicated code from being introduced.

Acceptance Criteria

• Run pylint similarities check on data/src/ to generate a comprehensive report of code duplication
• Refactor identified duplicated code by extracting common functionality into reusable components
• Ensure all existing tests continue to pass after refactoring changes
• Update pre-commit hook configuration to make pylint similarities check blocking (remove || true)
• Update relevant code comments and documentation to reflect structural changes

Additional context

• Use the existing pylint similarities configuration: pylint --disable=all --enable=similarities --score=no data/src/
• Prioritize changes that provide the most significant reduction in technical debt