Refactor: Externalize prediction loading logic for database compatibility

[sandrohuni]

Summary

Refactors prediction loading logic out of model classes into a centralized prediction_loader module, enabling database compatibility and improving separation of concerns.

Resolves #58

Changes

New Module: lt_forecasting/scr/prediction_loader.py

Created centralized prediction loading utilities:

• load_predictions_from_filesystem() - Load from CSV files
• load_predictions_from_dataframe() - Load from pre-loaded DataFrames (database-ready)
• apply_area_conversion() - Apply area-based unit conversions
• handle_duplicate_predictions() - Handle duplicate (date, code) pairs
• standardize_prediction_columns() - Ensure consistent column naming

Updated Model Classes

SciRegressor:

• Added base_predictors and base_model_names optional parameters to __init__
• Updated __load_lr_predictors__() to use external predictions when provided
• Added deprecation warning for internal file-based loading
• Maintains full backward compatibility

BaseMetaLearner:

• Added base_predictors and base_model_names optional parameters to __init__
• Updated __load_base_predictors__() to use external predictions when provided
• Handles Q_ prefix normalization (external uses Q_, internal doesn't)
• Added deprecation warning for internal file-based loading
• Maintains full backward compatibility

Tests

• Added 24 comprehensive unit tests for prediction_loader module
• All existing tests pass (191 passed, 1 skipped)
• No breaking changes to existing functionality

Usage

Old Way (still works with deprecation warning)

```python
model = SciRegressor(data, static_data, general_config, model_config,
feature_config, path_config)
```

New Way (recommended)

```python
from lt_forecasting.scr.prediction_loader import (
load_predictions_from_filesystem,
apply_area_conversion
)

Load predictions externally

base_preds, model_names = load_predictions_from_filesystem(
path_config["path_to_lr_predictors"],
join_type="inner"
)

Apply conversions if needed

base_preds = apply_area_conversion(base_preds, static_data, model_names)

Pass to model

model = SciRegressor(
data, static_data, general_config, model_config,
feature_config, path_config,
base_predictors=base_preds,
base_model_names=model_names
)
```

Database-Ready Pattern

```python
from lt_forecasting.scr.prediction_loader import load_predictions_from_dataframe

Load from database

df = load_from_database(query="SELECT * FROM predictions WHERE ...")

Convert to standard format

base_preds, model_names = load_predictions_from_dataframe(
df, model_names=['model1', 'model2']
)

Use with model

model = BaseMetaLearner(
data, static_data, general_config, model_config,
feature_config, path_config,
base_predictors=base_preds,
base_model_names=model_names
)
```

Benefits

1. Database Compatibility: Can now load predictions from any source (DB, API, files)
2. Testability: Easy to mock predictions without file I/O
3. Maintainability: Single source of truth for loading logic
4. Flexibility: Support different storage backends
5. Performance: Can cache predictions across multiple model runs
6. Separation of Concerns: Models focus on modeling, not data loading

Backward Compatibility

✅ All existing code continues to work
✅ No breaking changes
✅ Deprecation warnings guide users to new pattern
✅ All 191 tests passing

Test Plan

• ✅ All unit tests pass (24 new + 167 existing)
• ✅ All functionality tests pass
• ✅ No regressions in model behavior
• ✅ Prediction loading works with both old and new patterns

Migration Path

1. Immediate: Existing code works unchanged (with deprecation warnings)
2. Short-term: Update scripts to use new prediction_loader module
3. Long-term: Remove internal loading methods in future major version

🤖 Generated with Claude Code