Skip to content

Add petrolib.data_qc_io: cleaning, scaling, filtering, signal, synth, IO (PR-A, final domain) - #157

Merged
ingehap merged 1 commit into
mainfrom
claude/petrophysics-library-merge-plan-txjfx9
Jul 10, 2026
Merged

ingehap merged 1 commit into
mainfrom
claude/petrophysics-library-merge-plan-txjfx9

Conversation

@ingehap

@ingehap ingehap commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

Canonical module PR (PR-A) for data_qc_io — the final domain of LIBRARY_MERGE_PLAN.md. Adds the petrolib/data_qc_io/ package with six submodules, each function grounded in a four-agent corpus survey of the srcYYYY_MM article forms:

  • clean — sentinels_to_nan (np.isclose matching, src2021_12/article01), impute_gaps (linear interp over NaN runs, src2018_12/article11), outlier_mask (zscore/iqr/mad with side='both'|'upper'|'lower'), despike (MAD·1.4826; replace='interp'|'median'|'nan'; the upper/median combination reproduces src2024_08/magnetic_permeability bit-exactly), closure_residual, renormalize, relative_discrepancy (the two corpus arrangements 2|a-b|/(a+b) and |a-b|/(0.5(a+b)) round identically in IEEE, so one canonical body serves both).
  • scale — normalize_to_reference (Shier two-point map; explicit endpoints, percentile default (5,95), or pct=None min/max — covers src2021_12 normalize_reference, src2016_12 histogram_normalize (opposite arg order, mapped by keyword), and src2015_04 histogram_scale), match_moments (src2016_12 affine_normalize).
  • filt — smooth (boxcar/gaussian/median with per-kind corpus edge conventions), moving_stat (centered-shrinking or causal windows; mean/std/var/mad/cv, all ddof=0; reproduces rolling_cv, moving_std, and moving_msd bit-exactly), tool_response (verbatim src2023_10/article_09 Gaussian vertical response incl. the odd-length |1 kernel), median_filter2d (generalized src2023_06 3×3 stack median), window_features (edge-padded sliding_window_view stacks of the induction papers), detect_bed_boundaries (verbatim src2023_08/article5 |gradient|·sqrt(var) local-max pick).
  • signal — snr_db (noise record or noise_std; guard reproduces the VMD +1e-30 convention), stack_gain, block_stack (block-mean, trailing remainder dropped), add_gaussian_noise (verbatim src2021_12/article01, sigma = fraction·mean|x|, deterministic default rng).
  • synth — blocky_log, log_suite (verbatim src2022_08 draw sequence; returns a curve dict + boundaries + facies), shifted_pair, gaussian_mixture_spectrum (the -0.5 standard-deviation form), sphere_pack_volume, disk_image (verbatim src2024_04 SEM draw sequence).
  • io — WellboreData, the Bradley et al. (2023) universal wellbore JSON container (verbatim semantics from src2023_12/bradley_wellbore_format.py, plus a from_json classmethod).

Registered data_qc_io in the PEP 562 lazy-load list of petrolib/__init__.py.

Scope decisions vs the plan sketch

Four planned files are deliberately omitted because the canon already exists:

  • units.py → petrolib.units.convert + petrolib.constants already cover the psi/bar/MPa/ft families from the same sources.
  • metrics.py → petrolib.ml_stats (rmse/mae/mape/r2_score/pearson_r) and its zscore/minmax scaling.
  • align.py → petrolib.depth_matching (xcorr_shift, dtw, shifts); detect_bed_boundaries lives in filt instead.
  • pay.py → petrolib.porosity_lithology (pay_flag, interval_thickness, 3-arg net_to_gross).

Verification

  • 37/37 bit-exact pre-checks against the live article functions (np.array_equal, incl. NaN-aware), covering every intended delegation target — including the subtle ones: np.var vs mean((x-mean)²) for the causal MSD, reshape-mean vs slice-mean block stacking, both relative-discrepancy arrangements, the full synth_log_suite/synthetic_sem_image RNG draw sequences, and byte-identical WellboreData.to_json output.
  • ruff check + ruff format --check clean; python3 -m mypy petrolib clean (45 source files).
  • python3 -m pytest tests/petrolib: 252 passed (12 new in tests/petrolib/test_data_qc_io.py).
  • No article files touched in this PR — adoption follows in era-chunked PR-Bs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B2BX1XL8422gebCU1oVoQC


Generated by Claude Code

Final plan domain (LIBRARY_MERGE_PLAN.md data_qc_io). Package with six
submodules, each grounded in a corpus survey of the srcYYYY_MM article
forms and pre-verified bit-exact against the delegation targets:

- clean: sentinels_to_nan, impute_gaps, outlier_mask (zscore/iqr/mad,
  sided), MAD despike (interp/median/nan), closure_residual, renormalize,
  relative_discrepancy (both corpus arrangements round identically)
- scale: normalize_to_reference (Shier two-point, percentile or min/max
  endpoints), match_moments
- filt: smooth (boxcar/gaussian/median), moving_stat (centered or causal;
  mean/std/var/mad/cv), tool_response (Gaussian vertical response),
  median_filter2d, window_features (edge-padded stacks),
  detect_bed_boundaries (|gradient|*sqrt(var) local maxima)
- signal: snr_db (noise record or noise_std, VMD guard), stack_gain,
  block_stack, add_gaussian_noise (sigma = fraction * mean|x|)
- synth: blocky_log, log_suite (verbatim CPA draw sequence), shifted_pair,
  gaussian_mixture_spectrum, sphere_pack_volume, disk_image (verbatim SEM
  draw sequence)
- io: WellboreData (Bradley et al. 2023 universal JSON container, plus
  from_json)

Deliberately not duplicated: unit conversions (petrolib.units), metrics
and zscore/minmax (petrolib.ml_stats), xcorr/DTW alignment
(petrolib.depth_matching), pay flags and net-to-gross
(petrolib.porosity_lithology).

37/37 bit-exact pre-checks against article bodies; ruff + mypy clean;
252 tests pass (12 new).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2BX1XL8422gebCU1oVoQC
@ingehap
ingehap marked this pull request as ready for review July 10, 2026 08:53
@ingehap
ingehap merged commit 6cc8deb into main Jul 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants