diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/README.md b/rnog_analysis_tools/data_monitoring/science_verification_analysis/README.md new file mode 100644 index 0000000..e053bcf --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/README.md @@ -0,0 +1,132 @@ +# Science Verification Analysis (SVA) + +## Purpose + +The Science Verification Analysis checks whether an RNO-G station is behaving as expected, using a set of automated diagnostic tests run over a chosen set of runs. Each test looks at a different failure mode (galactic-noise sensitivity, SNR/RMS stability, channel glitching, ADC block offsets, trigger rates) and reports a per-channel health verdict (`OK`, `!!`, or `X`). The output is a concise CSV summary for every channel plus a set of detailed plots and text/JSON reports for anyone who needs to dig into a flagged channel. + +The analysis is meant to be run regularly (e.g. after a station comes back online, or periodically to catch slow degradation) and to be usable by anyone on the collaboration, not just the analysis' original author — channel maps, thresholds, and plotting styles all live in config files rather than in code, and reference ("expected") values are calculated once per station from a known-good time period and then reused. + +There are two ways to read the underlying data: + +- **`monitoring.root`** (current, recommended): reads pre-computed per-event summary quantities (RMS, max amplitude, glitching test statistic, block offsets, spectra) from the station's monitoring stream. Fast, and works for all data taken with the monitoring pipeline. +- **`dataProviderRNOG`** (outdated/legacy): reads full waveforms via NuRadioReco for older runs that predate `monitoring.root`. Kept only for archival purposes — see the [[Legacy path](https://claude.ai/cowork/local_dbaff4ef-131f-445a-bc91-dabf4583340d#sva_dataproviderrnog-legacy)](#sva_dataproviderrnog-legacy) section below for its caveats. + +## Quick start / usage example + +Run the main analysis for station 14 over an explicit list of runs: + +```bash +python science_verification_analysis_main.py -st 14 --runs 260080 260090 260100 +``` + +Or over a run range, excluding a couple of bad runs: + +```bash +python science_verification_analysis_main.py -st 14 --run_range 260080 260150 --exclude-runs 260101 260102 +``` + +Or over a date range (run numbers are looked up from the RNO-G run table): + +```bash +python science_verification_analysis_main.py -st 14 --time_range 2026-04-26 2026-05-02 +``` + +`--runs`, `--run_range`, and `--time_range` are mutually exclusive — pick exactly one way to select runs. `-st`/`--station_id` and exactly one of these are required; `-ex`/`--exclude-runs` and `--debug_plot` (produce extra diagnostic plots) are optional. + +Use `--data_location` to choose where the raw data is read from and where results are written (default `desy`): + +- `desy` — reads from the DESY inbox (`/pnfs/ifh.de/acs/radio/diskonly/data/inbox/`), writes results under `/pnfs/ifh.de/acs/radio/diskonly/NuRadioMC/science_verification_analysis/`. +- `uchicago` — reads from the UChicago mirror (`/data/satellite`), writes results under `/data/sva`. +- any other value is treated as a custom base data path, with results written to a `results/` subdirectory under it. + +Each run of the script creates its own uniquely-named output directory (so concurrent/repeated runs never collide or overwrite each other): `/_station-_run-run_/`, containing: + +- `channel_health_summary/validation_summary_station_.csv` — the top-level per-channel verdict table. +- `detailed_results/` — per-test text/JSON files with the numbers behind each verdict (spectral results, SNR/RMS outlier details, glitching stats, block offset stats, failed-run report). +- `plots/standard_plots/`, `plots/failed_test_plots/`, `plots/other_debug_plots/` — plots are split into these three subdirectories: always-produced standard plots, and debug plots routed to `failed_test_plots` or `other_debug_plots` depending on whether the corresponding test passed for all channels. Please check the standard plots always and do not forget to check the failed plots in case any test fails. +- `logs/logging_science_verification_analysis_station_.log` — a full log of the run, including warnings about missing files, invalid timestamps, and borderline channels. + +> **Note:** the results base paths (`/pnfs/...`) are dCache-mounted storage. dCache enforces write-once semantics per file — a file can only be opened for writing once; a second `open(..., "w")` or `open(..., "a")` on the same path fails with `PermissionError: [Errno 1] Operation not permitted` even though the file exists and is nominally writable. Because of this, every output-writing function in `helper_functions/output_writer.py` builds its full content in memory first and performs exactly one `open()`/write/close per file — don't reintroduce per-item append/reopen loops when editing these. + +## Analysis overview + +Each station is configured (in `config_station.json`) with a `daq_type` of either `radiant` or `didaq`, which determines the non-FORCE trigger names used throughout (`LT`/`RADIANT0`/`RADIANT1` for RADIANT stations vs. `DIDAQ_DEEP_PHASED`/`DIDAQ_SURF_UP`/`DIDAQ_SURF_DOWN` for DIDAQ stations) and which summary-CSV builder is used (`create_result_csv_file()` vs. `create_result_csv_file_didaq()`). Glitching and block-offset analysis (steps 6–7 below) only run for RADIANT stations — DIDAQ stations skip them entirely. + +For a given station and set of runs, `science_verification_analysis_main.py` runs the following steps in order: + +1. **Read data** — `read_multiple_runs()` reads `monitoring.root` and `headers.root` for every requested run, validates them against each other (matching event numbers, station/run IDs, non-overlapping trigger types), and concatenates everything into one combined dataset. Runs with missing or inconsistent files are skipped and reported rather than crashing the whole analysis. +2. **Spectral (galactic noise) test** — normalizes surface-channel spectra to a reference band and checks, in several frequency bands, whether upward-facing channels show more galactic excess than downward-facing ones (as expected). +3. **SNR stability (z-score) test** — compares each channel's log-SNR distribution for the current runs against a previously-computed reference (mean/std/k-value per channel) and flags statistically significant outlier events. +4. **RMS/Vrms modality & tail test** — uses a KDE of the RMS distribution per channel to check it's unimodal (not bimodal/flat, which would suggest a mis-biased or noisy channel), and separately checks for excess skew/tails. +5. **RMS stability test** — same z-score approach as SNR, plus a relative-median-shift metric across runs, combined into an overall stability decision per channel. +6. **Glitching test** (RADIANT only) — a one-sided binomial test on how often each channel's glitching test statistic is triggered, against an expected background rate. +7. **Block offset test** (RADIANT only) — summarizes each channel's ADC block-offset statistics (mean/median/std/IQR/P99) and compares them to reference values; informational only (not counted in the overall channel-health verdict). +8. **Trigger rate plots** — plots trigger rates over time per trigger type, no pass/fail verdict. +9. **Summary CSV** — `create_result_csv_file()` (RADIANT) or `create_result_csv_file_didaq()` (DIDAQ) combines the SNR, spectral, RMS, RMS-stability, and (for RADIANT) glitching verdicts into one `OK`/`!!`/`X` per channel and writes the summary table. + +Steps 2–7 each have their own config file (thresholds, band definitions, KDE parameters, etc. — see below) so they can be tuned without touching analysis code, and their own reference/"expected value" script under `expected_values/` to (re-)generate reference numbers for a new station or after a known-good re-calibration period. + +## Directory structure + +### `science_verification_analysis_main.py` +The entry point described above. Also home to `setup_logging()`. `REFERENCE_DIR` and `CONFIG_DIR` are script-relative constants so the script can be run from anywhere, but the per-run output directories (`plots/`, `detailed_results/`, `channel_health_summary/`, `logs/`) are computed inside `if __name__ == "__main__":` under a uniquely-named directory whose base path depends on `--data_location` (see Quick start above). + +### `config_files_sva/` +All tunable parameters, one JSON file per topic (plus one small Python helper). Editing these does not require touching analysis code: + +- `config_station.json` — channel maps (`all_channels`, `surface_channels`, `deep_channels`, `upward_channels`, `downward_channels`, `vpol_channels`, `hpol_channels`, `phased_array_channels`, `reference_channels`, `reference_channels_galaxy`) under a `default_config`, with a `station_specific_adjustments` block keyed by station ID (currently station 14 has a different channel layout). +- `config_helper.py` — `get_station_config(station_id, default_config, station_specific_adjustments)` merges the default config with any station-specific overrides. +- `config_spectral_analysis.json` — frequency bands used for the galactic-noise test (`galactic_excess`, plus a few RFI bands), the normalization band, and the significance thresholds (`alpha_spec`, `ci_threshold_spec`, `log_ratio_thresholds_spec`). +- `config_rms.json` — parameters for the KDE modality test (`bandwidth`, `grid_points`, `peak_prominence`, `height_threshold`), the tail/skewness test (percentiles, `extreme_k`, minimum event count), and the human-readable reporting thresholds (`strong_skew`, `extreme_skew`, etc.). +- `config_glitching.json` — the binomial test parameters for the glitching test (`alpha`, expected background `pvalue`, confidence level, and the CI thresholds that separate "weak"/"moderate"/"strong" excessive glitching). +- `config_block_offsets.json` — acceptable median/IQR block-offset limits. +- `config_plotting.py` — `set_plot_style()` centralizes all matplotlib `rcParams` (fonts, tick sizes, line widths, dpi) plus a shared 24-color palette (`COLORS`), so every plot in the analysis looks consistent. + +### `analysis_functions_sva/` +The statistics/physics behind each test, split by topic and independent of I/O: + +- `spectral_analysis_sva.py` — `normalize_channels()`, `find_amplitude_ratio_in_band[_specific_bkg]()`, `excess_info_from_ratio[_specific_bkg]()`, `validate_excess_in_bands()`. Computes upward/downward amplitude ratios per frequency band and classifies them as NO/WEAK/MODERATE/STRONG excess (method differs slightly for monitoring vs. dataProviderRNOG data). +- `z_score_analysis_sva.py` — shared statistics machinery used by both the SNR and RMS tests: log-parameter statistics, z-scores against a reference mean/std, k-value derivation (`find_k_value`), outlier flagging/detail extraction, saving/loading reference values to/from JSON (including the metadata block), and a rolling-window z-score variant. +- `vrms_analysis_sva.py` — `calculate_vrms()` (from raw waveforms, dataProviderRNOG path) and `get_rms_per_trigger_monitoring()` (from precomputed RMS, monitoring path) both split values by trigger type; `kde_modality()` and `tail_fraction_and_trimmed_skew_two_sided()` implement the modality/tail tests; `report_vrms_characteristics()` turns those into human-readable labels. +- `vrms_stability_analysis_sva.py` — `get_rms_per_run()`, `relative_median_shift()` (pairwise relative shift in median RMS between runs), and `decision_metric()`, which combines outlier fraction, largest z-score excess, and median-shift into the final `OK`/`!!`/`X` RMS-stability verdict. Also contains a block of commented-out, currently-unused helpers (Wasserstein-distance and linear-regression-based stability metrics) kept for possible future use. +- `glitching_analysis_sva.py` — `binomtest_glitch_fraction()`, a one-sided binomial test per channel classifying glitching as NO/WEAK/MODERATE/STRONG excessive. +- `block_offsets_analysis_sva_monitoring.py` — `get_force_block_offsets_monitoring()`, `block_offset_statistics_monitoring()` (mean/median/std/IQR/P99/P95), and a violin-plot helper, for the monitoring.root path. +- `block_offsets_analysis_sva_dataproviderrnog.py` — equivalent block-offset functions for the legacy dataProviderRNOG path (computes offsets before *and* after removal, since that path removes block offsets from the waveform). + +### `monitoring_data_functions_sva/` +- `get_monitoring_data_uproot.py` — everything related to reading `monitoring.root`/`headers.root` with `uproot`: low-level readers (`get_event_info_from_monitoring_file`, `get_run_summary_from_monitoring_file`, `get_info_from_header_file`), trigger-type assignment and consistency checks (`assign_trigger_types`, `check_event_numbers_according_to_trigger_types`), SNR calculation, and the main `read_multiple_runs()` entry point that loops over runs, validates and concatenates everything, and reports failed runs instead of raising. + +### `helper_functions/` +Small, reusable utilities that don't belong to a specific test: + +- `config_helper.py` is imported from `config_files_sva/` (see above) but conceptually lives here too. +- `output_writer.py` — every "write results to disk" function used by the main script: failed-run CSV, spectral results text file, SNR/RMS outlier-detail text files, RMS modality text files, glitching results text file, block-offset results text file, and `create_result_csv_file()`/`create_result_csv_file_didaq()`, which assemble the final per-channel summary CSV and compute the combined `channel_health()` verdict. Because the results filesystem (dCache/pnfs) only allows a file to be opened for writing once, `write_spectral_results()` takes the results for *all* surface channels at once and writes them in a single `open()` call, rather than being called once per channel. +- `read_rnog_runtable.py` — `read_rnog_runtable()`, a thin wrapper around `rnog_data.runtable` used to turn a `--time_range` into a list of run numbers. + +### `plotting_functions_sva/` +One module per plot family, all consuming already-computed arrays/dicts (no analysis logic lives here): + +- `plotting_sva_spectrum.py` — time-integrated surface/deep spectrum plots (normalized and unnormalized). +- `plotting_sva_snr.py` — SNR-vs-time plots per channel, including the outlier flags and z-score/k-value bands; also `choose_day_interval()`, a small helper that picks a sensible tick spacing based on the time range. +- `plotting_sva_vrms.py` — RMS/Vrms-vs-time plots (all triggers together and per trigger), the z-score single-trigger plot, rolling-mean plots, and the relative-median-shift heatmap (`create_heatmap_plot`). +- `plotting_sva_glitch.py` — glitching violin plots and the 99th-percentile-glitching-over-time plot. +- `plotting_sva_debug.py` — extra diagnostic plots (amplitude-ratio distributions, raw/z-scored SNR distributions, RMS distributions with the KDE overlay) enabled with `--debug_plot`. +- `plotting_sva_trigger_rate.py` — trigger-rate-over-time and trigger-rate-heatmap plots. + +### `expected_values/` +Scripts to (re-)generate the reference ("expected") values that the SNR and RMS stability tests compare against, plus the values themselves: + +- `expected_snr_values.py` / `expected_rms_values.py` — standalone CLIs (same run-selection arguments as the main script — `-st`/`--station_id`, one of `--runs`/`--run_range`/`--time_range`, `-ex`/`--exclude-runs`, `--data_location` — plus `--save-values`, and no `--debug_plot`) that read a known-stable period for a station and compute per-channel k-value/mean/std, writing them to `expected_snr/expected_snr_values_station.json` / `expected_rms/expected_rms_station.json`. Like the main script, they read `daq_type` from `config_station.json` so they work for both RADIANT and DIDAQ stations (`expected_rms_values.py` maps all four trigger names accordingly; `expected_snr_values.py` only needs `daq_type` for `read_multiple_runs()`/`choose_trigger_type_header()` since it only ever uses the FORCE trigger, whose name is shared between the two DAQ types). Each JSON file carries a `metadata` block (station ID, run range, excluded runs, event count, trigger type, start/end time, and any comment — including an automatic note when a k-value hit the min/max cap) alongside the `values` block, so a reference file is self-documenting about how and when it was produced. Unlike the main script, these write to fixed, script-relative directories (`plots_reference/`, `logs_reference/`, `results_reference/`, created next to the script) rather than a dynamic per-run directory, since they're run occasionally/manually rather than as part of the regular monitoring pipeline. +- `expected_snr/`, `expected_rms/` — one combined JSON file per station (not split by parameter as in earlier versions). +- `expected_block_offsets/` — one JSON per station with the reference block-offset statistics (median/IQR/P95/P99, in both ADC counts and mV) and the simulation settings used to derive them. +- `outdated/` — reference-value scripts and results for the legacy dataProviderRNOG method (`expected_snr_values_dataproviderrnog.py`, `expected_rms_values_dataproviderrnog.py`). Per its own README: "These scripts won't be updated anymore." + +### `sva_dataproviderrnog/` (legacy) +- `read_rnog_data_nuradio.py` — reads full waveforms via NuRadioReco's `readRNOGDataMattak` from `combined.root` files. +- `science_verification_analysis_dataprovider.py` — the original, pre-monitoring.root version of the full analysis, kept only for stations/periods without `monitoring.root` files. Its own module docstring spells out the caveats: no reference-value files exist for it by default (you'd need to run the `expected_values/outdated` scripts first and update the file paths), and its RMS analysis is done in ADC units, which is explicitly flagged as incorrect. Prefer `science_verification_analysis_main.py` whenever `monitoring.root` is available. + +### `outdated/` (top-level, legacy) +Pre-JSON-config versions of the station/trigger-rate configuration (`config_station.py`, `trigger_rate.py`), superseded by `config_files_sva/config_station.json`. Kept for reference only; not imported by any current analysis code. + +### Output directories (created at runtime, not checked in) +For `science_verification_analysis_main.py`, output directories are created dynamically per run under a `--data_location`-dependent base path — see [Quick start](#quick-start--usage-example) above, not next to the script. For the `expected_values/` scripts, `plots_reference/`, `logs_reference/`, `results_reference/` are created next to the script itself. The `channel_health_summary/`, `detailed_results/`, `logs/`, and `plots/` directories present directly under this folder are historical outputs from before the dynamic per-run output scheme was introduced; they aren't written to by the current version of `science_verification_analysis_main.py`. None of these need to exist beforehand — every script creates its output directories with `os.makedirs(..., exist_ok=True)`. \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_dataproviderrnog.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_dataproviderrnog.py new file mode 100644 index 0000000..e701dc8 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_dataproviderrnog.py @@ -0,0 +1,98 @@ +import os + +import numpy as np +import matplotlib.pyplot as plt +from NuRadioReco.modules.RNO_G.channelBlockOffsetFitter import fit_block_offsets +from NuRadioReco.utilities import units + +def get_block_offsets_after_removal(trace_arr, event_info, channel_list, sampling_rate=2.4*units.GHz): + force_mask = event_info["triggerType"] == "FORCE" + trace_arr_force = trace_arr[:, force_mask, :] + n_force_events = trace_arr_force.shape[1] + + fit_blocks = np.zeros((len(channel_list), n_force_events, 16)) + + for ch in channel_list: + traces_force_ch = trace_arr_force[ch] + offsets_ch = np.array([fit_block_offsets(trace, sampling_rate=sampling_rate) for trace in traces_force_ch]) + fit_blocks[ch] = offsets_ch + + return fit_blocks + +def get_block_offsets_before_removal(block_offset_arr, event_info, channel_list): + force_mask = event_info["triggerType"] == "FORCE" + block_offset_arr_force = block_offset_arr[:, force_mask, :] + n_force_events = block_offset_arr_force.shape[1] + + fit_blocks = np.zeros((len(channel_list), n_force_events, 16)) + + for ch in channel_list: + offsets_ch = block_offset_arr_force[ch] + fit_blocks[ch] = offsets_ch + + return -fit_blocks # - because in chp the negative is stored + +def block_offset_statistics(fit_blocks_before, fit_blocks_after, channel_list): + stats = {} + for ch in channel_list: + before_ch = fit_blocks_before[ch].flatten() + after_ch = fit_blocks_after[ch].flatten() + stats[ch] = { + "before_mean": np.nanmean(before_ch), + "before_median": np.nanmedian(before_ch), + "before_std": np.nanstd(before_ch), + "after_mean": np.nanmean(after_ch), + "after_median": np.nanmedian(after_ch), + "after_std": np.nanstd(after_ch), + "iqr_before": np.nanpercentile(before_ch, 75) - np.nanpercentile(before_ch, 25), + "iqr_after": np.nanpercentile(after_ch, 75) - np.nanpercentile(after_ch, 25), + } + + removal_fraction = 1 - (np.median((after_ch)) / np.median((before_ch))) if np.median((before_ch)) != 0 else np.nan + stats[ch]["removal_fraction"] = removal_fraction + iqr_reduction_fraction = 1 - (stats[ch]["iqr_after"] / stats[ch]["iqr_before"]) if stats[ch]["iqr_before"] != 0 else np.nan + stats[ch]["iqr_reduction_fraction"] = iqr_reduction_fraction + return stats + + +def plot_block_offsets_violin_before_after_comparison(fit_blocks_before, fit_blocks_after, channel_list, station_id, run_label, save_location): + fit_blocks_before_flat = fit_blocks_before.reshape(len(channel_list), -1) + fit_blocks_after_flat = fit_blocks_after.reshape(len(channel_list), -1) + + fig, ax = plt.subplots(figsize=(12, 6)) + positions_before = np.array(channel_list) + positions_after = np.array(channel_list) + 0.2 + + parts_before = ax.violinplot(fit_blocks_before_flat.T, positions=positions_before, showextrema=True, showmedians=True, vert=False, side="high", widths=1.8) + parts_after = ax.violinplot(fit_blocks_after_flat.T, positions=positions_after, showextrema=True, showmedians=True, vert=False, side="high", widths=1.8) + + for pc in parts_before["bodies"]: + pc.set_facecolor("lightblue") + pc.set_edgecolor("blue") + pc.set_alpha(0.7) + + for pc in parts_after["bodies"]: + pc.set_facecolor("crimson") + pc.set_edgecolor("darkred") + pc.set_alpha(0.7) + + parts_before["cmedians"].set_color("black") + parts_before["cmedians"].set_linewidth(1.8) + parts_after["cmedians"].set_color("black") + parts_after["cmedians"].set_linewidth(1.8) + + ax.set_xlabel("Fitted block offset [V]") + ax.set_ylabel("Channel") + ax.grid(True, alpha=0.3) + + n_force_events = fit_blocks_before.shape[1] + ax.plot(np.nan, np.nan, label=f"{n_force_events} FORCE triggers", color="k") + ax.plot(np.nan, np.nan, label="removed block offsets", color="C0") + ax.plot(np.nan, np.nan, label="after removal", color="C1") + ax.legend(loc="best") + + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"block_offset_violin_comparison_{station_id}_{run_label}.pdf")) + plt.close(fig) + + diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_monitoring.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_monitoring.py new file mode 100644 index 0000000..f8722ea --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/block_offsets_analysis_sva_monitoring.py @@ -0,0 +1,52 @@ +import os +import matplotlib.pyplot as plt +from NuRadioReco.utilities import units +import numpy as np + +def get_force_block_offsets_monitoring(block_offset_arr, force_mask): + print(f"Original block_offset_arr shape: {block_offset_arr.shape}") + # block_offset_arr has the shape (n_ch, n_events) + block_offset_arr_force = block_offset_arr[:, force_mask] + print(f"Filtered block_offset_arr_force shape: {block_offset_arr_force.shape}") + return block_offset_arr_force + +def block_offset_statistics_monitoring(block_offset_arr_force, channel_list): + stats = {} + for ch in channel_list: + offsets_ch = block_offset_arr_force[ch] # No need to flatten since already 2D with shape (n_events, n_blocks) + stats[ch] = { + "mean": np.nanmean(offsets_ch), + "median": np.nanmedian(offsets_ch), + "std": np.nanstd(offsets_ch), + "iqr": np.nanpercentile(offsets_ch, 75) - np.nanpercentile(offsets_ch, 25), + "p99": np.nanpercentile(offsets_ch, 99), + "p95": np.nanpercentile(offsets_ch, 95), + } + return stats + +def plot_block_offsets_violin_monitoring(block_offset_arr_force, channel_list, station_id, run_label, save_location): + # block_offset_arr_force has the shape (n_ch, n_events) + fig, ax = plt.subplots(figsize=(12, 6)) + + positions = np.array(channel_list) + violin_plot = ax.violinplot(block_offset_arr_force.T, positions=positions, showextrema=True, showmedians=True, vert=False, side="high", widths=1.8) + + for pc in violin_plot['bodies']: + pc.set_facecolor("lightblue") + pc.set_edgecolor("blue") + pc.set_alpha(0.7) + + violin_plot["cmedians"].set_color("k") + violin_plot["cmedians"].set_linewidth(1.8) + + ax.set_xlabel("Block Offsets") + ax.set_ylabel("Channel") + ax.grid(True, alpha=0.3) + + n_force_events = block_offset_arr_force.shape[1] + ax.plot(np.nan, np.nan, label=f"{n_force_events} FORCE triggers", color="k") + ax.legend(loc="best") + + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"block_offset_violin_monitoring_{station_id}_{run_label}.pdf")) + plt.close(fig) diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/glitching_analysis_sva.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/glitching_analysis_sva.py new file mode 100644 index 0000000..3ad3e38 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/glitching_analysis_sva.py @@ -0,0 +1,47 @@ +import numpy as np +from scipy.stats import binomtest + + +def binomtest_glitch_fraction(glitch_arr, channel_list, config_glitching=None): + '''Perform binomial test on glitch fractions for each channel (with p0=0.1 - can be adjusted from the config_glitching.py).''' + if config_glitching is None: + config_glitching = {} + + alpha = config_glitching.get("alpha", 0.01) + pvalue = config_glitching.get("pvalue", 0.1) + ci_level = config_glitching.get("ci_level", 0.99) + strong_ci_threshold = config_glitching.get("strong_ci_threshold", 0.3) + moderate_ci_threshold = config_glitching.get("moderate_ci_threshold", 0.2) + + glitch_info = {} + n_events = glitch_arr.shape[1] + + for ch in channel_list: + glitch_ch = glitch_arr[ch] + n_glitches = np.sum(glitch_ch > 0) + + result = binomtest(n_glitches, n_events, p=pvalue, alternative="greater") + pval = result.pvalue + statistic = result.statistic + confidence_interval = result.proportion_ci(confidence_level=ci_level) + + if pval > alpha: + validation = "NO EXCESSIVE GLITCHING" + else: + if confidence_interval.low > strong_ci_threshold: + validation = "STRONG EXCESSIVE GLITCHING" + elif confidence_interval.low > moderate_ci_threshold: + validation = "MODERATE EXCESSIVE GLITCHING" + else: + validation = "WEAK EXCESSIVE GLITCHING" + + glitch_info[ch] = { + "n_glitches": int(n_glitches), + "n_events": int(n_events), + "pval": float(pval), + "confidence_interval": (float(confidence_interval.low), float(confidence_interval.high)), + "glitch_fraction": float(n_glitches / n_events), + "validation": validation, + } + + return glitch_info \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/spectral_analysis_sva.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/spectral_analysis_sva.py new file mode 100644 index 0000000..68b128e --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/spectral_analysis_sva.py @@ -0,0 +1,216 @@ +import numpy as np +from NuRadioReco.utilities import units +from scipy.stats import binomtest +import logging + +logger = logging.getLogger(__name__) + +#### Normalize surface channel spectra to the average of the down channels in the reference frequency band (500-650 MHz) +def normalize_channels(spec_arr, frequencies, down_channels, up_channels, normalization_band = None): + '''Normalize all surface channel spectra to the average of the down channels in the reference frequency band. The reference band is 500-650 MHz by default''' + + if normalization_band is None: + normalization_band = {} + + f_low = normalization_band.get("freq_min", 500)*units.MHz + f_high= normalization_band.get("freq_max", 650)*units.MHz + + # spec_arr shape: (n_channels, n_events, n_freqs) + freq_mask = (frequencies >= f_low) & (frequencies <= f_high) + spec_arr = np.copy(spec_arr) + + # Use only down channels to define the reference + down = spec_arr[down_channels] + down_band_avg = np.mean(down[:, :, freq_mask], axis=2) # (n_down, n_events) + ref_band_avg = np.mean(down_band_avg, axis=0) # (n_events,) + + all_surface_channels = up_channels + down_channels + all_surface_spectra = spec_arr[all_surface_channels] + + ch_band_avg = np.mean(all_surface_spectra[:, :, freq_mask], axis=2) # (n_ch, n_events) + scale_factors = ref_band_avg[np.newaxis, :] / ch_band_avg # (n_ch, n_events) + + all_surface_spectra_norm = all_surface_spectra * scale_factors[:, :, np.newaxis] + spec_arr[all_surface_channels] = all_surface_spectra_norm + + return spec_arr, scale_factors + +def normalize_channels_new( + spec_arr, + frequencies, + down_channels, + up_channels, + normalization_band=None +): + ''' + Normalize each surface-channel spectrum by its own average value within + the reference frequency band. + + The normalization is performed separately for each channel and event. + By default, the reference frequency band is 500-650 MHz. + ''' + + if normalization_band is None: + normalization_band = {} + + f_low = normalization_band.get("freq_min", 300) * units.MHz + f_high = normalization_band.get("freq_max", 350) * units.MHz + + # spec_arr shape: (n_channels, n_events, n_freqs) + freq_mask = (frequencies >= f_low) & (frequencies <= f_high) + + if not np.any(freq_mask): + raise ValueError( + f"No frequency bins found between " + f"{f_low / units.MHz:.1f} and {f_high / units.MHz:.1f} MHz" + ) + + spec_arr = np.copy(spec_arr) + + all_surface_channels = np.concatenate(( + np.asarray(up_channels, dtype=int), + np.asarray(down_channels, dtype=int) + )) + + all_surface_spectra = spec_arr[all_surface_channels] + + # Average each channel separately within the normalization band + ch_band_avg = np.nanmean( + all_surface_spectra[:, :, freq_mask], + axis=2 + ) # (n_surface_channels, n_events) + + scale_factors = np.full_like(ch_band_avg, np.nan, dtype=float) + + valid = np.isfinite(ch_band_avg) & (ch_band_avg != 0) + + np.divide( + 1.0, + ch_band_avg, + out=scale_factors, + where=valid + ) + + all_surface_spectra_norm = ( + all_surface_spectra * scale_factors[:, :, np.newaxis] + ) + + spec_arr[all_surface_channels] = all_surface_spectra_norm + + return spec_arr, scale_factors + +def find_amplitude_ratio_in_band(freqs, norm_spec_arr, upward_channels, downward_channels, reference_channels, freq_min, freq_max): + '''Find normalized amplitude ratio of upward vs downward channels in a given frequency band. This will be replaced by lab measurements in the future.''' + freq_mask = (freqs >= freq_min) & (freqs <= freq_max) + ratio_list = [] + spectrum_list = [] + + ref_specs = np.stack([norm_spec_arr[ch][:, freq_mask] for ch in reference_channels], axis=0) + ref_spectra_per_ch = np.median(ref_specs, axis=2) # (n_ref_ch, n_events) + ref_spec = np.median(ref_spectra_per_ch, axis=0) # (n_events,) + + for ch in upward_channels + downward_channels: + masked_spec = norm_spec_arr[ch][:, freq_mask] # (n_events, n_freqs) + masked_spec_med = np.median(masked_spec, axis=1) # (n_events,) + + spec_ratio = masked_spec_med / ref_spec + ratio_list.append(spec_ratio) + spectrum_list.append(masked_spec_med) + + return np.asarray(ratio_list), np.asarray(spectrum_list) + + +def find_amplitude_ratio_in_band_specific_bkg(freqs, norm_spec_arr, upward_channels, downward_channels, **bandconfig): + '''Find normalized amplitude ratio of upward vs downward channels in specific frequency bands. Backgrouns were defined using wiki page: https://radio.uchicago.edu/wiki/index.php/Features_observed_in_data\n + bandconfig should be a dict with keys as band names and values as dicts with keys: freq_min, freq_max, reference_channels. Example: {"galactic_excess": {"freq_min": 80*units.MHz, "freq_max": 120*units.MHz, "reference_channels": reference_channels_galaxy}, ...}''' + + ratio_arr_dict = {} + for band_name, band_info in bandconfig.items(): + freq_min = band_info["freq_min"]*units.MHz + freq_max = band_info["freq_max"]*units.MHz + reference_channels = band_info["reference_channels"] + + ratio_arr, _ = find_amplitude_ratio_in_band(freqs, norm_spec_arr, upward_channels, downward_channels, reference_channels, freq_min, freq_max) + ratio_arr_dict[band_name] = ratio_arr + + return ratio_arr_dict + +def excess_info_from_ratio(ratio_arr, band_name, alpha, ci_thresholds, use_monitoring, log_ratio_thresholds): + '''Calculate excess information from amplitude ratios in frequency bands. Different methods are applied depending on whether monitoring data is used or not. + If monitoring data is used, log ratio thresholds are applied for excess validation. If not, a binomial test is applied to calculate p-value and confidence intervals for excess validation.''' + + log_ratio = np.log10(np.asarray(ratio_arr)) + median_log_ratio = np.median(log_ratio) + mean_log_ratio = np.mean(log_ratio) + + if use_monitoring: + std_log_ratio = np.std(log_ratio) + n_runs = len(log_ratio) + + no_excess = log_ratio_thresholds.get("no_excess", 0.08) + weak_excess = log_ratio_thresholds.get("weak_excess", 0.12) + moderate_excess = log_ratio_thresholds.get("moderate_excess", 0.16) + + if median_log_ratio < no_excess: + validation = "NO EXCESS" + elif median_log_ratio < weak_excess: + validation = f"WEAK EXCESS" + elif median_log_ratio < moderate_excess: + validation = f"MODERATE EXCESS" + else: + validation = f"STRONG EXCESS" + return { + "median_log_ratio": median_log_ratio, + "mean_log_ratio": mean_log_ratio, + "std_log_ratio": std_log_ratio, + "n_runs": n_runs, + "validation": validation + } + + else: + frac_pos_to_neg = np.mean(log_ratio > 0)/np.mean(log_ratio < 0) if np.mean(log_ratio < 0) != 0 else np.inf + + k = np.sum(log_ratio > 0) + n = int(log_ratio.size) + result = binomtest(k, n, p=0.5, alternative="greater") + pval = result.pvalue + statistic = result.statistic + confidence_interval = result.proportion_ci(confidence_level=0.99) + + if pval > alpha: + validation = "NO EXCESS" + else: + if confidence_interval.low > ci_thresholds[1]: + validation = f"STRONG EXCESS" + elif confidence_interval.low > ci_thresholds[0]: + validation = f"MODERATE EXCESS" + else: + validation = f"WEAK EXCESS" + + return { + "median_log_ratio": median_log_ratio, + "mean_log_ratio": mean_log_ratio, + "99% CI": confidence_interval, + "statistic - k over n": statistic, + "frac_pos_to_neg": frac_pos_to_neg, + "pval": pval, + "validation": validation + } + +def excess_info_from_ratio_specific_bkg(ratio_arr_dict, alpha=0.005, ci_thresholds=(0.6, 0.75), use_monitoring=False, log_ratio_thresholds=None): + '''Calculate excess information from amplitude ratios in specific frequency bands.''' + excess_info_dict = {} + for band_name, ratio_arr in ratio_arr_dict.items(): + excess_info = excess_info_from_ratio(ratio_arr, band_name, alpha, ci_thresholds, use_monitoring, log_ratio_thresholds) + excess_info_dict[band_name] = excess_info + + return excess_info_dict + +# Validation +def validate_excess_in_bands(excess_info_dict): + '''Validate excess in different frequency bands based on excess information.''' + validation_dict = {} + for band_name, excess_info in excess_info_dict.items(): + validation_dict[band_name] = excess_info["validation"] + + return validation_dict \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_analysis_sva.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_analysis_sva.py new file mode 100644 index 0000000..ab192e4 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_analysis_sva.py @@ -0,0 +1,204 @@ +import numpy as np +from scipy.signal import find_peaks +from scipy.stats import gaussian_kde, skew +import os +import logging + +logger = logging.getLogger(__name__) + +# For dataProbiderRNOG +def calculate_vrms(trace_arr, event_info): + '''Calculate Vrms for each channel and event according to trigger types.''' + vrms_arr = np.std(trace_arr, axis=2) # (n_channels, n_events) + + force_mask = event_info["triggerType"] == "FORCE" + radiant0_mask = event_info["triggerType"] == "RADIANT0" + radiant1_mask = event_info["triggerType"] == "RADIANT1" + lt_mask = event_info["triggerType"] == "LT" + + vrms_arr_force = vrms_arr[:, force_mask] + vrms_arr_radiant0 = vrms_arr[:, radiant0_mask] + vrms_arr_radiant1 = vrms_arr[:, radiant1_mask] + vrms_arr_lt = vrms_arr[:, lt_mask] + + return vrms_arr, vrms_arr_force, vrms_arr_radiant0, vrms_arr_radiant1, vrms_arr_lt + +def get_rms_per_trigger_monitoring(rms_arr, force_mask, lt_mask, radiant0_mask, radiant1_mask): + '''Get RMS values for each channel and event according to trigger types for monitoring data.''' + + # Still named Vrms to keep consistent with the dataProviderRNOG case, but these are actually RMS values calculated in the monitoring pipeline, not Vrms calculated from traces as in the dataProviderRNOG case. The analysis functions will be the same for both cases, just the input values are different. + vrms_arr = rms_arr # (n_channels, n_events) + vrms_arr_force = rms_arr[:, force_mask] + vrms_arr_lt = rms_arr[:, lt_mask] + vrms_arr_radiant0 = rms_arr[:, radiant0_mask] + vrms_arr_radiant1 = rms_arr[:, radiant1_mask] + + return vrms_arr,vrms_arr_force, vrms_arr_radiant0, vrms_arr_radiant1, vrms_arr_lt + +def kde_modality(vrms_arr, channel_list, kde_modality_config=None): + '''Calculate KDE and modality for Vrms distributions.''' + + if kde_modality_config is None: + kde_modality_config = {} + + bandwidth = kde_modality_config.get("bandwidth", None) + grid_points = kde_modality_config.get("grid_points", 512) + peak_prominence = kde_modality_config.get("peak_prominence", 0.01) + height_threshold = kde_modality_config.get("height_threshold", 0.05) + + modality_dict = {} + + for ch in channel_list: + vrms_ch = vrms_arr[ch] + vrms_ch = vrms_ch[~np.isnan(vrms_ch)] + + kde = gaussian_kde(vrms_ch, bw_method=bandwidth) + vrms_min = np.min(vrms_ch) + vrms_max = np.max(vrms_ch) + vrms_grid = np.linspace(vrms_min, vrms_max, grid_points) + kde_values = kde(vrms_grid) + + # Adjust prominence to be relative to the KDE range + abs_prom = peak_prominence * (np.max(kde_values) - np.min(kde_values)) + + peaks, properties = find_peaks(kde_values, prominence=abs_prom, height=height_threshold*np.max(kde_values)) + + modality_dict[ch] = { + "kde": kde, + "vrms_grid": vrms_grid, + "kde_values": kde_values, + "n_peaks": len(peaks), + "peaks": peaks, + "prominences": properties["prominences"], + } + + return modality_dict + +def tail_fraction_and_trimmed_skew_two_sided(vrms_arr, channel_list, skewness_config=None): + '''Calculate tail fraction and two-sided trimmed skewness for Vrms distributions.''' + tail_dict = {} + + if skewness_config is None: + skewness_config = {} + + lower_percentile = skewness_config.get("lower_percentile", 25) + upper_percentile = skewness_config.get("upper_percentile", 75) + extreme_k = skewness_config.get("extreme_k", 2) + min_events_for_skew = skewness_config.get("min_events_for_skew", 30) + max_tail_frac_for_trimmed_skew = skewness_config.get("max_tail_frac_for_trimmed_skew", 0.05) + + for ch in channel_list: + vrms_ch = vrms_arr[ch] + vrms_ch = vrms_ch[~np.isnan(vrms_ch)] + n_events = len(vrms_ch) + + full_skew = skew(vrms_ch, bias=False) + + q1, q3 = np.percentile(vrms_ch, [lower_percentile, upper_percentile]) + iqr = q3 - q1 + + lower_bound = q1 - extreme_k * iqr + upper_bound = q3 + extreme_k * iqr + + high_mask = vrms_ch > upper_bound + high_frac = np.mean(high_mask) + + low_mask = vrms_ch < lower_bound + low_frac = np.mean(low_mask) + + core_high = vrms_ch[~high_mask] + skew_trim_high = skew(core_high, bias=False) if len(core_high) > min_events_for_skew and high_frac < max_tail_frac_for_trimmed_skew else np.nan + + core_low = vrms_ch[~low_mask] + skew_trim_low = skew(core_low, bias=False) if len(core_low) > min_events_for_skew and low_frac < max_tail_frac_for_trimmed_skew else np.nan + + tail_dict[ch] = { + "n_events": n_events, + "full_skew": full_skew, + "high_tail_fraction": high_frac, + "low_tail_fraction": low_frac, + "trimmed_skew_high": skew_trim_high, + "trimmed_skew_low": skew_trim_low, + } + + return tail_dict + +def report_vrms_characteristics(modality_dict, tail_dict, channel_list, report_config=None): + + if report_config is None: + report_config = {} + + strong_skew = report_config.get("strong_skew", 0.3) + extreme_skew = report_config.get("extreme_skew", 0.5) + delta_skew_min = report_config.get("delta_skew_min", 0.25) + rare_max_high_frac = report_config.get("rare_max_high_frac", 0.01) + rare_max_low_frac = report_config.get("rare_max_low_frac", 0.01) + mod_max_high_frac = report_config.get("mod_max_high_frac", 0.05) + mod_max_low_frac = report_config.get("mod_max_low_frac", 0.05) + + modality_channels = {} + tail_label_channels = {} + + for ch in channel_list: + n_peaks = modality_dict[ch]["n_peaks"] + if n_peaks == 0: + modality = "flat/noisy" + elif n_peaks == 1: + modality = "unimodal" + elif n_peaks == 2: + modality = "bimodal" + else: + modality = f"multimodal ({n_peaks} peaks)" + + full_skew = tail_dict[ch]["full_skew"] + high_frac = tail_dict[ch]["high_tail_fraction"] + low_frac = tail_dict[ch]["low_tail_fraction"] + skew_trim_h = tail_dict[ch]["trimmed_skew_high"] + skew_trim_l = tail_dict[ch]["trimmed_skew_low"] + + if np.isnan(full_skew): + tail_label = "no significant tails" + tail_frac = None + else: + if not np.isnan(skew_trim_h): + dskew_h = full_skew - skew_trim_h + else: + dskew_h = 0 + if not np.isnan(skew_trim_l): + dskew_l = full_skew - skew_trim_l + else: + dskew_l = 0 + + if 0 < high_frac < rare_max_high_frac and full_skew > extreme_skew and dskew_h > delta_skew_min: + tail_label = "rare high extremes" + tail_frac = high_frac + + elif 0 < low_frac < rare_max_low_frac and full_skew < -extreme_skew and dskew_l < -delta_skew_min: + tail_label = "rare low extremes" + tail_frac = low_frac + + elif full_skew > strong_skew: + if high_frac < mod_max_high_frac: + tail_label = "moderate high skew" + else: + tail_label = "bulk high skew" + tail_frac = high_frac + + elif full_skew < -strong_skew: + if low_frac < mod_max_low_frac: + tail_label = "moderate low skew" + else: + tail_label = "bulk low skew" + tail_frac = low_frac + + else: + tail_label = "no significant tails" + tail_frac = None + + # output summary + if tail_frac is not None: + tail_label += f" (fraction: {tail_frac:.3f})" + modality_channels[ch] = modality + tail_label_channels[ch] = tail_label + + return modality_channels, tail_label_channels \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_stability_analysis_sva.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_stability_analysis_sva.py new file mode 100644 index 0000000..1f8c5f4 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/vrms_stability_analysis_sva.py @@ -0,0 +1,189 @@ +import numpy as np +import logging +from scipy.stats import mannwhitneyu as mw_test +import os +from scipy.stats import wasserstein_distance, linregress + +logger = logging.getLogger(__name__) + +def get_rms_per_run(rms_arr, run_no): + '''Get RMS values for each channel and run number.''' + run_no_unique = np.unique(run_no) + rms_arr_per_run_dict = {} + for run in run_no_unique: + run_mask = run_no == run + rms_arr_per_run_dict[run] = rms_arr[:, run_mask] + return rms_arr_per_run_dict + +def relative_median_shift(rms_arr_per_run_dict, channel_list): + '''Calculate relative median shift of RMS distributions across runs for each channel.''' + run_nos = sorted(rms_arr_per_run_dict.keys()) + median_shift_results = {} + + for ch in channel_list: + medians = [np.median(rms_arr_per_run_dict[run][ch]) for run in run_nos] + median_shift_matrix = np.full((len(run_nos), len(run_nos)), np.nan) + np.fill_diagonal(median_shift_matrix, 0) + for i in range(len(run_nos)): + for j in range(i+1,len(run_nos)): + shift = 2*(medians[j] - medians[i]) / (medians[i] + medians[j]) if (medians[i] + medians[j]) != 0 else np.nan + median_shift_matrix[i, j] = shift + median_shift_matrix[j, i] = -shift + median_shift_results[int(ch)] = {"medians": [float(m) for m in medians], "median_shift_matrix": median_shift_matrix.tolist()} + return median_shift_results + +### Currently not used, but can be used for future analysis if needed +# def wasserstein_distance_per_run(rms_arr_per_run_dict, channel_list): +# '''Calculate Wasserstein distance between RMS distributions of different runs for each channel.''' +# run_nos = sorted(rms_arr_per_run_dict.keys()) +# wasserstein_results = {} + +# for ch in channel_list: +# distance_matrix = np.full((len(run_nos), len(run_nos)), np.nan) +# np.fill_diagonal(distance_matrix, 0) +# for i in range(len(run_nos)): +# for j in range(i+1, len(run_nos)): +# dist = wasserstein_distance(rms_arr_per_run_dict[run_nos[i]][ch], rms_arr_per_run_dict[run_nos[j]][ch]) +# distance_matrix[i, j] = float(dist) +# distance_matrix[j, i] = float(dist) +# wasserstein_results[int(ch)] = {"distance_matrix": distance_matrix.tolist()} + +# return wasserstein_results + +# def wasserstein_statistics(wasserstein_results, channel_list): +# '''Calculate statistics of Wasserstein distances for each channel.''' +# wasserstein_stats = {} +# for ch in channel_list: +# distance_matrix = np.array(wasserstein_results[int(ch)]["distance_matrix"]) +# overall_max_distance = np.nanmax(distance_matrix) +# diagonal_matrix = np.diag(distance_matrix, k=1) + +# median_global_distance = np.nanmedian(distance_matrix) +# median_diagonal_distance = np.nanmedian(diagonal_matrix) +# mean_global_distance = np.nanmean(distance_matrix) +# mean_diagonal_distance = np.nanmean(diagonal_matrix) +# max_global_distance = np.nanmax(distance_matrix) +# max_diagonal_distance = np.nanmax(diagonal_matrix) +# std_global_distance = np.nanstd(distance_matrix) +# std_diagonal_distance = np.nanstd(diagonal_matrix) +# roughness = std_global_distance / mean_global_distance if mean_global_distance != 0 else np.nan +# wasserstein_stats[int(ch)] = { +# "overall_max_distance": float(overall_max_distance), +# "median_global_distance": float(median_global_distance), +# "median_diagonal_distance": float(median_diagonal_distance), +# "mean_global_distance": float(mean_global_distance), +# "mean_diagonal_distance": float(mean_diagonal_distance), +# "max_global_distance": float(max_global_distance), +# "max_diagonal_distance": float(max_diagonal_distance), +# "std_global_distance": float(std_global_distance), +# "std_diagonal_distance": float(std_diagonal_distance), +# "max_ratio": float(max_global_distance / max_diagonal_distance) if max_diagonal_distance != 0 else np.nan, +# "roughness": float(roughness), +# "fluctuation_index": float(std_global_distance / mean_global_distance) if mean_global_distance != 0 else np.nan , +# "jump_ratio": float(max_diagonal_distance / median_diagonal_distance) if median_diagonal_distance != 0 else np.nan, +# "global_p90": float(np.nanpercentile(distance_matrix, 90)), +# "diagonal_p90": float(np.nanpercentile(diagonal_matrix, 90)), +# } +# return wasserstein_stats + +# def linregress_rolling_mean(times, rolling_mean, channel_list): +# '''Perform linear regression on the rolling mean values over time for each channel.''' +# slope_dict = {} +# intercept_dict = {} +# r_value_dict = {} +# p_value_dict = {} +# std_err_dict = {} +# intercept_std_err_dict = {} + +# times = times.astype("datetime64[s]").astype(np.float64) # Convert to seconds since epoch for linregress +# times_rel = times - times.min() # Use relative time to avoid numerical issues with large values +# times_rel_hours = times_rel / 3600 # Convert to hours for better interpretability of slope + +# for ch in channel_list: +# res = linregress(times_rel_hours[1:], rolling_mean[ch][1:]) +# slope_dict[ch] = res.slope +# intercept_dict[ch] = res.intercept +# r_value_dict[ch] = res.rvalue +# p_value_dict[ch] = res.pvalue +# std_err_dict[ch] = res.stderr +# intercept_std_err_dict[ch] = res.intercept_stderr + +# return slope_dict, intercept_dict, r_value_dict, p_value_dict, std_err_dict, intercept_std_err_dict + +# def write_linregress_results(slope_dict, intercept_dict, r_value_dict, p_value_dict, std_err_dict, intercept_std_err_dict, station_id, run_label, trigger_name, results_dir): +# '''Write linear regression results to a txt file.''' +# results_file = os.path.join(results_dir, f"linear_regression_results_rolling_mean_{trigger_name}_station{station_id}_{run_label}.txt") +# with open(results_file, "w") as f: +# f.write(f"Linear Regression Results for Rolling Mean of Vrms - Station {station_id}, Trigger {trigger_name}, Runs {run_label}\n") +# for ch in slope_dict.keys(): +# f.write(f"Channel {ch}:\n") +# f.write(f" Slope: {slope_dict[ch]} ± {std_err_dict[ch]} ADC/hours \n") +# f.write(f" Intercept: {intercept_dict[ch]} ± {intercept_std_err_dict[ch]} ADC\n") +# f.write(f" R-value: {r_value_dict[ch]}\n") +# f.write(f" R-squared: {r_value_dict[ch]**2}\n") +# f.write(f" P-value: {p_value_dict[ch]}\n\n") +# logger.info(f"Linear regression results saved to {results_file}") + +def decision_metric(outlier_details, relative_median_shift, n_events_force, channels): + rms_results = {} + + for ch in channels: + outlier_ch_info = outlier_details.get(ch, []) + n_out = len(outlier_ch_info) + frac_out = n_out / n_events_force if n_events_force > 0 else 0.0 + + deltas = np.array( + [abs(o.get("z_minus_k", 0.0)) for o in outlier_ch_info], + dtype=float + ) + + max_delta = np.nanmax(deltas) if n_out > 0 else 0.0 + n_large_delta = np.sum(deltas >= 5.0) if n_out > 0 else 0 + frac_large_delta = n_large_delta / n_events_force if n_events_force > 0 else 0.0 + + median_shift_matrix = np.asarray(relative_median_shift[ch]["median_shift_matrix"]) + abs_shift_matrix = np.abs(median_shift_matrix) + + q95_rel_median_shift = np.nanquantile(abs_shift_matrix, 0.95) + max_rel_median_shift = np.nanmax(abs_shift_matrix) + + if max_delta >= 10: + if q95_rel_median_shift >= 0.1 or frac_out >= 0.002 or frac_large_delta >= 0.0002: + rms_value = "X" + else: + rms_value = "!!" + + elif max_delta >= 5: + if q95_rel_median_shift >= 0.10 or frac_out >= 0.004 or n_large_delta >= 10 or frac_large_delta >= 0.0002: + rms_value = "X" + else: + rms_value = "!!" + + elif max_delta >= 3: + if q95_rel_median_shift >= 0.10 or frac_out >= 0.01 or frac_large_delta >= 0.0004: + rms_value = "X" + elif q95_rel_median_shift >= 0.05 or frac_out >= 0.002 or frac_large_delta >= 0.0002: + rms_value = "!!" + else: + rms_value = "OK" + + else: + if q95_rel_median_shift >= 0.10 or frac_out >= 0.01 or frac_large_delta >= 0.0004: + rms_value = "X" + elif q95_rel_median_shift >= 0.05 or frac_out >= 0.004 or frac_large_delta >= 0.0002: + rms_value = "!!" + else: + rms_value = "OK" + + rms_results[ch] = { + "decision": rms_value, + "n_outliers": n_out, + "frac_outliers": frac_out, + "max_delta": max_delta, + "n_large_delta": int(n_large_delta), + "frac_large_delta": frac_large_delta, + "q95_rel_median_shift": q95_rel_median_shift, + "max_rel_median_shift": max_rel_median_shift, + } + + return rms_results \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/z_score_analysis_sva.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/z_score_analysis_sva.py new file mode 100644 index 0000000..394d6bb --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/analysis_functions_sva/z_score_analysis_sva.py @@ -0,0 +1,249 @@ +''' Z-score analysis for SNR and RMS values for RNO-G Science Verification Analysis. ''' + +import numpy as np +import logging +from scipy.stats import skew +import os +import json +import pandas as pd +from astropy.time import Time + +logger = logging.getLogger(__name__) + +def calculate_statistics_log_paramater(parameter_arr): + '''Calculate log10 statistics (mean, median, std, mean-median) for a given parameter array (e.g. SNR or RMS values).''' + log_parameter_arr = np.zeros((len(parameter_arr), len(parameter_arr[0]))) + for ch in range(len(parameter_arr)): + parameter_arr_ch = parameter_arr[ch] + log_parameter_arr_ch = np.log10(parameter_arr_ch) + log_parameter_arr[ch, :] = log_parameter_arr_ch + + log_mean_dict = {} + log_median_dict = {} + log_std_dict = {} + log_difference_dict = {} + + for ch in range(len(log_parameter_arr)): + log_mean_dict[ch] = np.mean(log_parameter_arr[ch]) + log_median_dict[ch] = np.median(log_parameter_arr[ch]) + log_std_dict[ch] = np.std(log_parameter_arr[ch]) + log_difference_dict[ch] = np.mean(log_parameter_arr[ch]) - np.median(log_parameter_arr[ch]) + + return log_parameter_arr, log_mean_dict, log_median_dict, log_std_dict, log_difference_dict + +def calculate_z_score_parameter(parameter_arr, ref_mean_dict, ref_std_dict, channel_list): + '''Calculate the z-score for a given parameter array (e.g. SNR or RMS values) given mean and standard deviation lists for each channel.''' + z_score_arr = np.zeros((len(parameter_arr), len(parameter_arr[0]))) + for ch in channel_list: + parameter_arr_ch = parameter_arr[ch] + mean_ch = ref_mean_dict[ch] + std_ch = ref_std_dict[ch] + z_score_arr_ch = (parameter_arr_ch - mean_ch) / std_ch + z_score_arr[ch,:] = z_score_arr_ch + + return z_score_arr + +def symmetry_metrics_channel_z_score(z_score): + return { + "mean": np.mean(z_score), + "median": np.median(z_score), + "skew": skew(z_score, bias=False), + "p_pos_3": np.mean(z_score > 3), + "p_neg_3": np.mean(z_score < -3), + "p_pos_5": np.mean(z_score > 5), + "p_neg_5": np.mean(z_score < -5), + } + +def symmetry_metrics_z_score(z_score): + metrics = {} + for ch in range(len(z_score)): + z_score_ch = z_score[ch] + metrics[f"ch_{ch}"] = symmetry_metrics_channel_z_score(z_score_ch) + return metrics + +#### For finding expected values and k-values for SNR and RMS monitoring +def find_k_value(z_score_log, channel_list, quantile=0.999): + '''Find the k-value corresponding to the given reference z-score array and quantile.''' + k_ch_list = {} + for ch in channel_list: + z_score_ch = z_score_log[ch] + k_ch = np.quantile(np.abs(z_score_ch), quantile) + k_ch_list[ch] = k_ch + return k_ch_list + +def save_values_json(k_values_log, mean_values_log, std_values_log, filename, SCRIPT_DIR, metadata=None): + '''Save the reference values as a JSON file.''' + if not os.path.isabs(filename): + filepath = os.path.join(SCRIPT_DIR, filename) + else: + filepath = filename + + # Track capped k-values + capped_k_values_high = {f"Ch{ch}": float(k_values_log[ch]) for ch in k_values_log if float(k_values_log[ch]) >= 5} + capped_k_values_low = {f"Ch{ch}": float(k_values_log[ch]) for ch in k_values_log if float(k_values_log[ch]) <= 3} + + # Add metadata + if metadata is None: + metadata = {} + else: + metadata = metadata.copy() + + comments = [] + if capped_k_values_high: + comments.append(f"k values above 5 are set to 4: {capped_k_values_high}") + if capped_k_values_low: + comments.append(f"k values below 3 are set to 3: {capped_k_values_low}") + + if comments: + metadata["comment"] = "\n".join(comments) + + output = { + "metadata": metadata, + "values": { + ch: { + "k_value": float(k_values_log[ch]) if 3 < float(k_values_log[ch]) < 5 else (4.0 if float(k_values_log[ch]) > 5 else 3.0), + "mean": float(mean_values_log[ch]), + "std": float(std_values_log[ch]), + } for ch in k_values_log + } + } + with open(filepath, "w") as f: + json.dump(output, f, indent=4) + + print(f"Expected values saved to {filepath}.") + +#### For loading expected values from JSON files +def load_values_json(script_dir, filename): + ''' Load reference values from a JSON file.''' + if not os.path.isabs(filename): + filepath = os.path.join(script_dir, filename) + else: + filepath = filename + with open(filepath, "r") as f: + data = json.load(f) + + k_values = {int(ch): float(info["k_value"]) for ch, info in data["values"].items()} + mean_values = {int(ch): float(info["mean"]) for ch, info in data["values"].items()} + std_values = {int(ch): float(info["std"]) for ch, info in data["values"].items()} + + # Report metadata if available + if "metadata" in data: + logger.info("Metadata for loaded values:") + logger.info(f"Reference values for the analysis were calculated using the following metadata:\n") + for key, value in data["metadata"].items(): + logger.info(f"- {key}: {value}") + else: + logger.info("No metadata found in the loaded reference JSON file.") + + return k_values, mean_values, std_values + +def outlier_flag(z_score_log, k_values_log, channel_list): + '''Flag outlier events based on the k-values for each channel.''' + flag = np.zeros((len(channel_list), len(z_score_log[0])), dtype=bool) + for ch in channel_list: + flag[ch, :] = np.abs(z_score_log[ch]) > k_values_log[ch] + + return flag + +def find_outlier_details(z_score_log, k_values_log, flag, channel_list, run_no, event_number): + '''Find details of outlier events for each channel.''' + outlier_details = {} + + # Sanity check: + if flag.shape[1] != len(run_no) or flag.shape[1] != len(event_number): + raise ValueError(f"Length of run_no and event_number arrays must match the number of events in the z_score_log and flag arrays. run_no length: {len(run_no)}, event_number length: {len(event_number)}, z_score_log shape: {z_score_log.shape}, flag shape: {flag.shape}") + + for ch in channel_list: + outlier_indices = np.where(flag[ch, :])[0] + details_ch = [] + for idx in outlier_indices: + z_abs = np.abs(z_score_log[ch, idx]) + k_ch = k_values_log[ch] + delta = z_abs - k_ch + + details_ch.append({ + "run": int(run_no[idx]), + "eventNumber": int(event_number[idx]), + "z_abs": float(z_abs), + "k": float(k_ch), + "z_minus_k": float(delta), + }) + + outlier_details[ch] = details_ch + + return outlier_details + +#### Calculate z score from rolling mean and std +def calculate_z_score_rolling(parameter_arr, run_no, channel_list): + '''Calculate z-score using a rolling mean and std for each channel.''' + z_score_arr = np.zeros((len(parameter_arr), len(parameter_arr[0]))) + rolling_mean_arr = np.zeros((len(parameter_arr), len(parameter_arr[0]))) + rolling_std_arr = np.zeros((len(parameter_arr), len(parameter_arr[0]))) + + run_no_unique = np.unique(run_no) + n_runs = len(run_no_unique) + + for ch in channel_list: + parameter_arr_ch = parameter_arr[ch] + window_size = int(len(parameter_arr_ch)/n_runs) + parameter_series = pd.Series(parameter_arr_ch) + rolling_mean = parameter_series.rolling(window=window_size, min_periods=1).mean() + rolling_std = parameter_series.rolling(window=window_size, min_periods=1).std() + z_score_arr_ch = (parameter_series - rolling_mean) / rolling_std + z_score_arr[ch, :] = z_score_arr_ch.values + rolling_mean_arr[ch, :] = rolling_mean.values + rolling_std_arr[ch, :] = rolling_std.values + + return z_score_arr, rolling_mean_arr, rolling_std_arr + +def metadata_dict(station_id, first_run, last_run, times, trigger_type="FORCE", excluded_runs=None, comment=""): + '''Create a metadata dictionary to save with the expected values, containing station ID, run numbers and time period.''' + if isinstance(times, Time): + start_time = times.min().iso + end_time = times.max().iso + else: + start_time = str(np.min(times)) + end_time = str(np.max(times)) + + metadata = { + "station_id": station_id, + "run_range": f"{first_run} - {last_run}", + "excluded_runs": excluded_runs if excluded_runs else None, + "n_events": len(times), + "trigger_type": trigger_type, + "start_time": start_time, + "end_time": end_time, + "comment": comment + } + logger.info(f"Metadata for expected values: {metadata}") + return metadata + +def calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_trigger, times_trigger, trigger_type, excluded_runs, run_no,all_channels, comment=""): + '''Calculate expected values for a given trigger type.''' + vrms_mean = np.mean(vrms_arr_trigger, axis=1) + vrms_std = np.std(vrms_arr_trigger, axis=1) + + z_score = calculate_z_score_parameter(vrms_arr_trigger, vrms_mean, vrms_std, all_channels) + z_score_rolling, rolling_mean, rolling_std = calculate_z_score_rolling(vrms_arr_trigger, run_no, all_channels) + k_values = find_k_value(z_score, all_channels, quantile=0.999) + + metadata = metadata_dict(station_id, first_run, last_run, times_trigger, trigger_type=trigger_type, excluded_runs=excluded_runs, comment=comment) + + return z_score, z_score_rolling, k_values, vrms_mean, vrms_std, metadata, rolling_mean, rolling_std + +def outlier_details(z_score, k_values, channels, run_no, event_number_arr, trigger_label, max_k=4): + '''Find details of outlier events.''' + k_values = k_values.copy() + + for ch in channels: + if k_values[ch] > max_k: + logger.warning(f"Calculated k-value for channel {ch} for trigger {trigger_label} is {k_values[ch]:.2f}. Setting it to {max_k}.") + k_values[ch] = max_k + + flag_outliers = outlier_flag(z_score, k_values, channels) + outlier_details = find_outlier_details(z_score, k_values, flag_outliers, channels, run_no, event_number_arr) + + return flag_outliers, outlier_details + + + diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_block_offsets.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_block_offsets.json new file mode 100644 index 0000000..419ba2f --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_block_offsets.json @@ -0,0 +1,6 @@ +{ + "block_offset_limits": { + "median": 0.002, + "iqr": 0.005 + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_glitching.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_glitching.json new file mode 100644 index 0000000..b599747 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_glitching.json @@ -0,0 +1,9 @@ +{ + "config_glitching_values" : { + "alpha": 0.01, + "pvalue": 0.1, + "ci_level": 0.99, + "strong_ci_threshold": 0.3, + "moderate_ci_threshold": 0.2 + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_plotting.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_plotting.py new file mode 100644 index 0000000..e19b75a --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_plotting.py @@ -0,0 +1,70 @@ +import matplotlib as mpl +from cycler import cycler + +COLORS = [ + "#4477AA", # strong blue + "#EE6677", # coral / red + "#228833", # green + "#CCBB44", # yellow + "#66CCEE", # light blue + "#AA3377", # purple + + "#882255", # wine + "#44AA99", # teal + "#084C1F", # deep green + "#332288", # indigo + "#AA4499", # magenta + "#771122", # burgundy + + "#7089A1", # steel blue + "#DDCC77", # sand + "#B0D8EC", # pale blue + "#CC6677", # dusty red + "#999933", # olive + "#DCB43C", # warm yellow + + "#006699", # dark cyan + "#0099CC", # vivid cyan + "#9955AA", # lavender purple + "#55AA55", # bright green + "#CC7711", # warm orange + "#555555", # neutral gray +] + +def set_plot_style(): + mpl.rcParams.update({ + 'font.family': 'sans-serif', + 'font.sans-serif': ['Helvetica', 'Arial', 'DejaVu Sans'], + 'font.size': 17, + + 'axes.labelsize': 17, + 'axes.titlesize': 18, + 'axes.linewidth': 1.2, + 'axes.grid': False, + + 'axes.prop_cycle': cycler('color', COLORS), + + 'xtick.labelsize': 17, + 'ytick.labelsize': 17, + 'xtick.major.size': 6, + 'ytick.major.size': 6, + 'xtick.major.width': 1.2, + 'ytick.major.width': 1.2, + 'xtick.minor.size': 3, + 'ytick.minor.size': 3, + 'xtick.minor.visible': True, + 'ytick.minor.visible': True, + + 'lines.linewidth': 1.6, + 'lines.antialiased': True, + 'lines.markersize': 6, + + 'legend.fontsize': 14, + 'legend.frameon': False, + 'legend.handlelength': 1, + 'legend.borderpad': 0.3, + + 'figure.dpi': 120, + 'savefig.dpi': 300, + 'savefig.bbox': 'tight', +}) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_rms.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_rms.json new file mode 100644 index 0000000..a8faa8f --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_rms.json @@ -0,0 +1,26 @@ +{ + "kde_modality_function_parameters" : { + "bandwidth": null, + "grid_points": 512, + "peak_prominence": 0.01, + "height_threshold": 0.05 + }, + + "skewness_function_parameters" : { + "lower_percentile": 25, + "upper_percentile": 75, + "extreme_k": 2, + "min_events_for_skew": 30, + "max_tail_frac_for_trimmed_skew": 0.05 + }, + + "report_vrms_function_parameters" : { + "strong_skew": 0.3, + "extreme_skew": 0.5, + "delta_skew_min": 0.25, + "rare_max_high_frac": 0.01, + "rare_max_low_frac": 0.01, + "mod_max_high_frac": 0.05, + "mod_max_low_frac": 0.05 + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_spectral_analysis.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_spectral_analysis.json new file mode 100644 index 0000000..2d59757 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_spectral_analysis.json @@ -0,0 +1,24 @@ +{ + "spectral_bands" : { + "galactic_excess": {"freq_min": 80, "freq_max": 120}, + "freq_360_380MHz": {"freq_min": 360, "freq_max": 380}, + "freq_482_485MHz": {"freq_min": 482, "freq_max": 485}, + "freq_240_272MHz": {"freq_min": 240, "freq_max": 272} + }, + + "alpha_spec" : 0.005, + + "ci_threshold_spec" : [0.6, 0.75], + + "normalization_band" : { + "freq_min": 300, + "freq_max": 350 + }, + + "log_ratio_thresholds_spec" : { + "no_excess" : 0.08, + "weak_excess" : 0.12, + "moderate_excess" : 0.16 + + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_station.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_station.json new file mode 100644 index 0000000..4e2b4c9 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/config_files_sva/config_station.json @@ -0,0 +1,45 @@ +{ + "default_config" : { + "all_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 , 19 , 20, 21, 22, 23], + "surface_channels": [12, 13, 14, 15, 16, 17, 18 , 19 , 20], + "deep_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 21, 22, 23], + "upward_channels": [13, 16, 19], + "downward_channels": [12, 14, 15, 17, 18, 20], + "vpol_channels": [0, 1, 2, 3, 5, 6, 7, 9, 10, 22, 23], + "hpol_channels": [4, 8, 11, 21], + "phased_array_channels": [0, 1, 2, 3], + "reference_channels": [12, 14, 15, 17, 18, 20], + "reference_channels_galaxy": [12, 14, 15, 17, 18, 20], + "daq_type": "radiant" + }, + + "station_specific_adjustments" : { + "14" : { + "surface_channels": [12, 13, 14, 15, 16, 17, 18 , 19], + "deep_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23], + "upward_channels": [13, 15, 16, 18], + "downward_channels": [12, 14, 17, 19], + "vpol_channels": [0, 1, 2, 3, 5, 6, 7, 9, 10, 20, 22, 23], + "reference_channels": [12, 14, 17, 19], + "reference_channels_galaxy": [12, 14, 19] + }, + "25" : { + "surface_channels": [12, 13, 14, 15, 16, 17, 18 , 19], + "deep_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23], + "upward_channels": [16, 17, 18, 19], + "downward_channels": [12, 13, 14, 15], + "vpol_channels": [0, 1, 2, 3, 5, 6, 7, 9, 10, 20, 22, 23], + "reference_channels": [12, 14, 17, 19], + "reference_channels_galaxy": [12, 14], + "daq_type": "didaq" + } + }, + + "didaq_bits" : { + "DIDAQ_DEEP_PHASED": 8, + "DIDAQ_SURF_UP": 16, + "DIDAQ_SURF_DOWN": 32, + "DIDAQ_COINC0": 64, + "DIDAQ_COINC1": 128 + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_block_offsets/expected_block_offsets_station14.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_block_offsets/expected_block_offsets_station14.json new file mode 100644 index 0000000..5537b22 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_block_offsets/expected_block_offsets_station14.json @@ -0,0 +1,650 @@ +{ + "0": { + "median_adc_offset_counts": 6, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 3.567875768027096, + "iqr_adc_offset_mv": 1.1175441465817664, + "p99_adc_offset_counts": 10, + "p99_adc_offset_mv": 6.031680918811541, + "p95_adc_offset_counts": 9, + "p95_adc_offset_mv": 5.187333366845779, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 28.84781265258789, + "vrms_target_mv": 17.597165718078614, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "1": { + "median_adc_offset_counts": 3, + "iqr_adc_offset_counts": 1, + "median_adc_offset_mv": 1.7468358553452443, + "iqr_adc_offset_mv": 0.547150829262115, + "p99_adc_offset_counts": 5, + "p99_adc_offset_mv": 2.9531175360424515, + "p95_adc_offset_counts": 4, + "p95_adc_offset_mv": 2.5397240565485286, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 14.123920440673828, + "vrms_target_mv": 8.615591468811035, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "2": { + "median_adc_offset_counts": 8, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 4.805731208395195, + "iqr_adc_offset_mv": 1.5052701190201825, + "p99_adc_offset_counts": 13, + "p99_adc_offset_mv": 8.12434039614641, + "p95_adc_offset_counts": 11, + "p95_adc_offset_mv": 6.98705097099983, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 38.856407165527344, + "vrms_target_mv": 23.70240837097168, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "3": { + "median_adc_offset_counts": 7, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 4.079178377131942, + "iqr_adc_offset_mv": 1.2776963702263329, + "p99_adc_offset_counts": 11, + "p99_adc_offset_mv": 6.896064768359532, + "p95_adc_offset_counts": 10, + "p95_adc_offset_mv": 5.930716056493488, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 32.98191452026367, + "vrms_target_mv": 20.11896785736084, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "4": { + "median_adc_offset_counts": 4, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 2.437174111648892, + "iqr_adc_offset_mv": 0.7633813057846219, + "p99_adc_offset_counts": 7, + "p99_adc_offset_mv": 4.120170527457235, + "p95_adc_offset_counts": 6, + "p95_adc_offset_mv": 3.543406612826045, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 19.705602645874023, + "vrms_target_mv": 12.020417613983154, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "5": { + "median_adc_offset_counts": 7, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 4.441056144959512, + "iqr_adc_offset_mv": 1.39104515463129, + "p99_adc_offset_counts": 12, + "p99_adc_offset_mv": 7.507838094860368, + "p95_adc_offset_counts": 11, + "p95_adc_offset_mv": 6.456850020179571, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 35.90785217285156, + "vrms_target_mv": 21.903789825439453, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "6": { + "median_adc_offset_counts": 8, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 4.9564176398995095, + "iqr_adc_offset_mv": 1.5524687185358976, + "p99_adc_offset_counts": 14, + "p99_adc_offset_mv": 8.379083703571306, + "p95_adc_offset_counts": 12, + "p95_adc_offset_mv": 7.206133922563884, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 40.074771881103516, + "vrms_target_mv": 24.445610847473144, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "7": { + "median_adc_offset_counts": 8, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 5.083344279394263, + "iqr_adc_offset_mv": 1.592225182111168, + "p99_adc_offset_counts": 14, + "p99_adc_offset_mv": 8.593659837748959, + "p95_adc_offset_counts": 12, + "p95_adc_offset_mv": 7.390672520598317, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 41.10102844238281, + "vrms_target_mv": 25.071627349853514, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "8": { + "median_adc_offset_counts": 11, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 6.824866263861754, + "iqr_adc_offset_mv": 2.137711579739104, + "p99_adc_offset_counts": 19, + "p99_adc_offset_mv": 11.537793996661074, + "p95_adc_offset_counts": 16, + "p95_adc_offset_mv": 9.92267074208322, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 55.181983947753906, + "vrms_target_mv": 33.66101020812988, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "9": { + "median_adc_offset_counts": 10, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 6.22759765113757, + "iqr_adc_offset_mv": 1.950632744744798, + "p99_adc_offset_counts": 17, + "p99_adc_offset_mv": 10.52808011394773, + "p95_adc_offset_counts": 15, + "p95_adc_offset_mv": 9.054302108982789, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 50.35280990600586, + "vrms_target_mv": 30.715214042663572, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "10": { + "median_adc_offset_counts": 17, + "iqr_adc_offset_counts": 6, + "median_adc_offset_mv": 10.202598115766913, + "iqr_adc_offset_mv": 3.195698097556315, + "p99_adc_offset_counts": 28, + "p99_adc_offset_mv": 17.24802666299185, + "p95_adc_offset_counts": 24, + "p95_adc_offset_mv": 14.833553933886202, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 82.49240112304688, + "vrms_target_mv": 50.32036468505859, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "11": { + "median_adc_offset_counts": 10, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 6.340381647159755, + "iqr_adc_offset_mv": 1.9859593936466995, + "p99_adc_offset_counts": 18, + "p99_adc_offset_mv": 10.718747368354542, + "p95_adc_offset_counts": 15, + "p95_adc_offset_mv": 9.218278722477821, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 51.26471710205078, + "vrms_target_mv": 31.271477432250975, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "12": { + "median_adc_offset_counts": 9, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 5.472206584632213, + "iqr_adc_offset_mv": 1.7140261699536516, + "p99_adc_offset_counts": 15, + "p99_adc_offset_mv": 9.25105193855234, + "p95_adc_offset_counts": 13, + "p95_adc_offset_mv": 7.956039294056574, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 44.245147705078125, + "vrms_target_mv": 26.989540100097656, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "13": { + "median_adc_offset_counts": 9, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 5.373068427853325, + "iqr_adc_offset_mv": 1.682973724741296, + "p99_adc_offset_counts": 15, + "p99_adc_offset_mv": 9.083453690337544, + "p95_adc_offset_counts": 13, + "p95_adc_offset_mv": 7.811902361600802, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 43.443572998046875, + "vrms_target_mv": 26.500579528808593, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "14": { + "median_adc_offset_counts": 5, + "iqr_adc_offset_counts": 1, + "median_adc_offset_mv": 2.8296407244932165, + "iqr_adc_offset_mv": 0.8863112490980565, + "p99_adc_offset_counts": 8, + "p99_adc_offset_mv": 4.783655899110941, + "p95_adc_offset_counts": 7, + "p95_adc_offset_mv": 4.114013687888545, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 22.878864288330078, + "vrms_target_mv": 13.956107215881348, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "15": { + "median_adc_offset_counts": 7, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 4.158505226796583, + "iqr_adc_offset_mv": 1.302543439539655, + "p99_adc_offset_counts": 12, + "p99_adc_offset_mv": 7.030170963916959, + "p95_adc_offset_counts": 10, + "p95_adc_offset_mv": 6.046049336267323, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 33.62330627441406, + "vrms_target_mv": 20.51021682739258, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "16": { + "median_adc_offset_counts": 8, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 5.099307122977492, + "iqr_adc_offset_mv": 1.597225127055748, + "p99_adc_offset_counts": 14, + "p99_adc_offset_mv": 8.62064586117325, + "p95_adc_offset_counts": 12, + "p95_adc_offset_mv": 7.413880893460135, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 41.23009490966797, + "vrms_target_mv": 25.15035789489746, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "17": { + "median_adc_offset_counts": 5, + "iqr_adc_offset_counts": 1, + "median_adc_offset_mv": 3.2375839921664844, + "iqr_adc_offset_mv": 1.0140888513932707, + "p99_adc_offset_counts": 9, + "p99_adc_offset_mv": 5.47330536662676, + "p95_adc_offset_counts": 8, + "p95_adc_offset_mv": 4.707122266148204, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 26.177261352539062, + "vrms_target_mv": 15.968129425048827, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "18": { + "median_adc_offset_counts": 12, + "iqr_adc_offset_counts": 4, + "median_adc_offset_mv": 7.62238095050193, + "iqr_adc_offset_mv": 2.3875122812810616, + "p99_adc_offset_counts": 21, + "p99_adc_offset_mv": 12.886034358891925, + "p95_adc_offset_counts": 18, + "p95_adc_offset_mv": 11.082177074010728, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 61.63023376464844, + "vrms_target_mv": 37.59444259643555, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "19": { + "median_adc_offset_counts": 9, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 5.319824979722609, + "iqr_adc_offset_mv": 1.6662966015254241, + "p99_adc_offset_counts": 15, + "p99_adc_offset_mv": 8.99344285166999, + "p95_adc_offset_counts": 13, + "p95_adc_offset_mv": 7.734491730454553, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 43.01307678222656, + "vrms_target_mv": 26.237976837158204, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "20": { + "median_adc_offset_counts": 9, + "iqr_adc_offset_counts": 3, + "median_adc_offset_mv": 5.32787575595079, + "iqr_adc_offset_mv": 1.6688182974683912, + "p99_adc_offset_counts": 15, + "p99_adc_offset_mv": 9.00705311069086, + "p95_adc_offset_counts": 13, + "p95_adc_offset_mv": 7.746196751277222, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 43.07817077636719, + "vrms_target_mv": 26.277684173583985, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "21": { + "median_adc_offset_counts": 6, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 3.60734080142693, + "iqr_adc_offset_mv": 1.1299055402899967, + "p99_adc_offset_counts": 10, + "p99_adc_offset_mv": 6.098398625479244, + "p95_adc_offset_counts": 9, + "p95_adc_offset_mv": 5.244711565496412, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 29.16690444946289, + "vrms_target_mv": 17.79181171417236, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "22": { + "median_adc_offset_counts": 6, + "iqr_adc_offset_counts": 2, + "median_adc_offset_mv": 3.4324062660637735, + "iqr_adc_offset_mv": 1.0751118538668285, + "p99_adc_offset_counts": 10, + "p99_adc_offset_mv": 5.8026626280415945, + "p95_adc_offset_counts": 8, + "p95_adc_offset_mv": 4.99037430397098, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 27.752483367919922, + "vrms_target_mv": 16.929014854431152, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + }, + "23": { + "median_adc_offset_counts": 14, + "iqr_adc_offset_counts": 4, + "median_adc_offset_mv": 8.280904628277288, + "iqr_adc_offset_mv": 2.593777669801103, + "p99_adc_offset_counts": 23, + "p99_adc_offset_mv": 13.999303138432305, + "p95_adc_offset_counts": 20, + "p95_adc_offset_mv": 12.039604425375888, + "settings": { + "n_traces": 88225, + "n_samples": 2048, + "block_size": 128, + "fs_ghz": 2.4, + "vrms_target_adc": 66.95468139648438, + "vrms_target_mv": 40.84235565185547, + "band_mhz": [ + 0.1, + 0.8 + ], + "n_bits": 12, + "lsb_mv": 0.61, + "inject_percentage": 10, + "apply_adc": true, + "seed": 42 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station11.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station11.json new file mode 100644 index 0000000..c09e191 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station11.json @@ -0,0 +1,136 @@ +{ + "metadata": { + "station_id": 11, + "run_range": "260532 - 260649", + "excluded_runs": [ + 260537 + ], + "n_events": 76048, + "trigger_type": "FORCE", + "start_time": "2026-05-02T22:43:53", + "end_time": "2026-05-13T01:15:11", + "comment": "k values below 3 are set to 3: {'Ch21': 0.9003315100669806}" + }, + "values": { + "0": { + "k_value": 4.0, + "mean": 20.327800750732422, + "std": 0.730625569820404 + }, + "1": { + "k_value": 4.0, + "mean": 23.529115676879883, + "std": 0.69734787940979 + }, + "2": { + "k_value": 4.0, + "mean": 13.73550796508789, + "std": 0.42615818977355957 + }, + "3": { + "k_value": 4.0, + "mean": 12.3698148727417, + "std": 0.3659195005893707 + }, + "4": { + "k_value": 3.309363772392194, + "mean": 42.732627868652344, + "std": 1.0295854806900024 + }, + "5": { + "k_value": 4.0, + "mean": 42.56949234008789, + "std": 1.297562837600708 + }, + "6": { + "k_value": 4.0, + "mean": 37.0167121887207, + "std": 1.1780705451965332 + }, + "7": { + "k_value": 4.0, + "mean": 20.993803024291992, + "std": 1.3679813146591187 + }, + "8": { + "k_value": 3.3045135378837487, + "mean": 26.45004653930664, + "std": 0.6618129014968872 + }, + "9": { + "k_value": 4.0, + "mean": 32.5579948425293, + "std": 1.036858320236206 + }, + "10": { + "k_value": 3.852704254626729, + "mean": 34.28447341918945, + "std": 0.9568967819213867 + }, + "11": { + "k_value": 3.2894127931594768, + "mean": 27.672292709350586, + "std": 0.6999258995056152 + }, + "12": { + "k_value": 4.0, + "mean": 27.406274795532227, + "std": 15.151395797729492 + }, + "13": { + "k_value": 4.0, + "mean": 13.517607688903809, + "std": 1.5364950895309448 + }, + "14": { + "k_value": 4.0, + "mean": 17.001489639282227, + "std": 1.0191293954849243 + }, + "15": { + "k_value": 4.128826655387864, + "mean": 15.680522918701172, + "std": 0.42492395639419556 + }, + "16": { + "k_value": 4.0, + "mean": 14.133875846862793, + "std": 1.0125762224197388 + }, + "17": { + "k_value": 4.0, + "mean": 14.832402229309082, + "std": 0.652161717414856 + }, + "18": { + "k_value": 4.0, + "mean": 14.494171142578125, + "std": 0.5285356044769287 + }, + "19": { + "k_value": 4.0, + "mean": 15.335626602172852, + "std": 1.1616920232772827 + }, + "20": { + "k_value": 4.0, + "mean": 15.197550773620605, + "std": 0.5330485105514526 + }, + "21": { + "k_value": 3.0, + "mean": 19.622905731201172, + "std": 1.8187779188156128 + }, + "22": { + "k_value": 4.0, + "mean": 14.971864700317383, + "std": 0.4959898591041565 + }, + "23": { + "k_value": 4.0, + "mean": 21.956941604614258, + "std": 0.7579979300498962 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station12.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station12.json new file mode 100644 index 0000000..fa9ec6f --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station12.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 12, + "run_range": "260425 - 260518", + "excluded_runs": null, + "n_events": 25745, + "trigger_type": "FORCE", + "start_time": "2026-04-19T23:28:21", + "end_time": "2026-04-29T14:32:24", + "comment": "k values above 5 are set to 4: {'Ch13': 6.012516117096091, 'Ch19': 5.101125991821541, 'Ch22': 5.590720512390153}" + }, + "values": { + "0": { + "k_value": 3.3820942592620957, + "mean": 27.84986114501953, + "std": 0.6950475573539734 + }, + "1": { + "k_value": 3.3980813293457044, + "mean": 31.694591522216797, + "std": 0.7785265445709229 + }, + "2": { + "k_value": 3.3958904705047677, + "mean": 14.70964527130127, + "std": 0.38419967889785767 + }, + "3": { + "k_value": 3.3500452060699493, + "mean": 28.029321670532227, + "std": 0.6994935274124146 + }, + "4": { + "k_value": 3.242065904617317, + "mean": 47.03176498413086, + "std": 1.1068874597549438 + }, + "5": { + "k_value": 3.3543800582885774, + "mean": 40.598575592041016, + "std": 0.9727147817611694 + }, + "6": { + "k_value": 3.3379222679138207, + "mean": 34.6546516418457, + "std": 0.8411350250244141 + }, + "7": { + "k_value": 3.399773847579985, + "mean": 37.38059997558594, + "std": 0.9425309896469116 + }, + "8": { + "k_value": 3.2960206775665384, + "mean": 50.668853759765625, + "std": 1.1894290447235107 + }, + "9": { + "k_value": 3.2611116886138936, + "mean": 24.724153518676758, + "std": 0.618411123752594 + }, + "10": { + "k_value": 3.2193964042663743, + "mean": 77.39636993408203, + "std": 1.764606237411499 + }, + "11": { + "k_value": 3.309031341552787, + "mean": 35.54946517944336, + "std": 0.8292466402053833 + }, + "12": { + "k_value": 3.410605081558249, + "mean": 14.431412696838379, + "std": 0.3619319200515747 + }, + "13": { + "k_value": 4.0, + "mean": 14.18026351928711, + "std": 0.48038920760154724 + }, + "14": { + "k_value": 3.329072570800805, + "mean": 15.526840209960938, + "std": 0.3889968991279602 + }, + "15": { + "k_value": 3.394812593460112, + "mean": 15.922861099243164, + "std": 0.4026961922645569 + }, + "16": { + "k_value": 4.401855533599856, + "mean": 15.689773559570312, + "std": 0.5165758728981018 + }, + "17": { + "k_value": 3.2785102424621617, + "mean": 15.262103080749512, + "std": 0.39054903388023376 + }, + "18": { + "k_value": 3.326446134567281, + "mean": 16.168285369873047, + "std": 0.402048259973526 + }, + "19": { + "k_value": 4.0, + "mean": 13.834457397460938, + "std": 0.4023757576942444 + }, + "20": { + "k_value": 3.3545470371246506, + "mean": 14.236452102661133, + "std": 0.3600991368293762 + }, + "21": { + "k_value": 3.2507861557007107, + "mean": 35.510738372802734, + "std": 0.8564967513084412 + }, + "22": { + "k_value": 4.0, + "mean": 42.88630676269531, + "std": 8.352168083190918 + }, + "23": { + "k_value": 3.396711523056036, + "mean": 47.90447998046875, + "std": 1.1313132047653198 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station13.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station13.json new file mode 100644 index 0000000..12710f8 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station13.json @@ -0,0 +1,136 @@ +{ + "metadata": { + "station_id": 13, + "run_range": "260273 - 260383", + "excluded_runs": [ + 260348 + ], + "n_events": 62608, + "trigger_type": "FORCE", + "start_time": "2026-04-20T07:19:48", + "end_time": "2026-04-30T00:06:26", + "comment": "k values above 5 are set to 4: {'Ch2': 9.485967915534738}" + }, + "values": { + "0": { + "k_value": 3.0742022771835136, + "mean": 10.086199760437012, + "std": 0.8037351965904236 + }, + "1": { + "k_value": 3.2690687835216328, + "mean": 16.496919631958008, + "std": 0.42691534757614136 + }, + "2": { + "k_value": 4.0, + "mean": 15.155035972595215, + "std": 0.5579773783683777 + }, + "3": { + "k_value": 3.3025671114921478, + "mean": 21.088895797729492, + "std": 0.556576669216156 + }, + "4": { + "k_value": 3.2617390546798646, + "mean": 26.794395446777344, + "std": 0.6659552454948425 + }, + "5": { + "k_value": 3.2949084947109015, + "mean": 46.417022705078125, + "std": 1.0830636024475098 + }, + "6": { + "k_value": 3.2801302466392257, + "mean": 33.03574752807617, + "std": 0.8063671588897705 + }, + "7": { + "k_value": 3.3573470277786233, + "mean": 48.53841018676758, + "std": 1.1565738916397095 + }, + "8": { + "k_value": 3.2466449491977643, + "mean": 34.88233947753906, + "std": 0.8480294942855835 + }, + "9": { + "k_value": 3.4040321674346905, + "mean": 39.765132904052734, + "std": 1.115569829940796 + }, + "10": { + "k_value": 3.282618700504298, + "mean": 28.974565505981445, + "std": 0.70597904920578 + }, + "11": { + "k_value": 3.280792916059491, + "mean": 25.26082420349121, + "std": 0.6192628145217896 + }, + "12": { + "k_value": 3.3143753166198646, + "mean": 14.575870513916016, + "std": 0.3750319182872772 + }, + "13": { + "k_value": 4.449912326335681, + "mean": 13.599739074707031, + "std": 0.5025420784950256 + }, + "14": { + "k_value": 3.2059055397510408, + "mean": 15.183589935302734, + "std": 0.3917773365974426 + }, + "15": { + "k_value": 3.3212942161559815, + "mean": 13.958186149597168, + "std": 0.362388551235199 + }, + "16": { + "k_value": 3.413771431207634, + "mean": 13.182221412658691, + "std": 0.481511652469635 + }, + "17": { + "k_value": 3.2647109658718034, + "mean": 14.097810745239258, + "std": 0.3655184805393219 + }, + "18": { + "k_value": 3.3129263584613575, + "mean": 15.818865776062012, + "std": 0.4038383960723877 + }, + "19": { + "k_value": 3.024839308261776, + "mean": 12.2577486038208, + "std": 0.5580955147743225 + }, + "20": { + "k_value": 3.2379863312244397, + "mean": 16.305089950561523, + "std": 0.415583997964859 + }, + "21": { + "k_value": 3.300943661212899, + "mean": 35.69905471801758, + "std": 0.8526447415351868 + }, + "22": { + "k_value": 3.320789733886682, + "mean": 23.958187103271484, + "std": 0.5985663533210754 + }, + "23": { + "k_value": 3.3139623038768593, + "mean": 37.97699737548828, + "std": 0.9633722305297852 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station14.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station14.json new file mode 100644 index 0000000..138f183 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station14.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 14, + "run_range": "260157 - 260273", + "excluded_runs": null, + "n_events": 80820, + "trigger_type": "FORCE", + "start_time": "2026-05-02T23:46:20", + "end_time": "2026-05-13T00:13:09", + "comment": "k values below 3 are set to 3: {'Ch17': 2.6266605081558208}" + }, + "values": { + "0": { + "k_value": 3.2876625666618255, + "mean": 28.84781265258789, + "std": 0.710719883441925 + }, + "1": { + "k_value": 3.247021969079968, + "mean": 14.123920440673828, + "std": 0.3702831566333771 + }, + "2": { + "k_value": 3.2888835737705118, + "mean": 38.856407165527344, + "std": 0.9347506761550903 + }, + "3": { + "k_value": 3.2402560417652073, + "mean": 32.98191452026367, + "std": 0.8003800511360168 + }, + "4": { + "k_value": 3.30230705499649, + "mean": 19.705602645874023, + "std": 0.4900098443031311 + }, + "5": { + "k_value": 3.3436520941257366, + "mean": 35.90785217285156, + "std": 0.8664475679397583 + }, + "6": { + "k_value": 3.327472246885291, + "mean": 40.074771881103516, + "std": 0.949553906917572 + }, + "7": { + "k_value": 3.281896693468082, + "mean": 41.10102844238281, + "std": 0.9685581922531128 + }, + "8": { + "k_value": 3.265265287637689, + "mean": 55.181983947753906, + "std": 1.2795649766921997 + }, + "9": { + "k_value": 3.2579587237834855, + "mean": 50.35280990600586, + "std": 1.181276798248291 + }, + "10": { + "k_value": 3.2497685046195937, + "mean": 82.49240112304688, + "std": 1.870578408241272 + }, + "11": { + "k_value": 3.2965153298377894, + "mean": 51.26471710205078, + "std": 1.2116690874099731 + }, + "12": { + "k_value": 3.269932426214211, + "mean": 44.245147705078125, + "std": 1.0447485446929932 + }, + "13": { + "k_value": 4.0, + "mean": 43.443572998046875, + "std": 1.3145612478256226 + }, + "14": { + "k_value": 3.29345712041853, + "mean": 22.878864288330078, + "std": 0.5779138207435608 + }, + "15": { + "k_value": 3.682404065847394, + "mean": 33.62330627441406, + "std": 1.1097626686096191 + }, + "16": { + "k_value": 4.3527293252944865, + "mean": 41.23009490966797, + "std": 1.1193190813064575 + }, + "17": { + "k_value": 3.0, + "mean": 26.177261352539062, + "std": 1.2120496034622192 + }, + "18": { + "k_value": 4.332928720474197, + "mean": 61.63023376464844, + "std": 1.6671212911605835 + }, + "19": { + "k_value": 3.3471208152770835, + "mean": 43.01307678222656, + "std": 1.0275160074234009 + }, + "20": { + "k_value": 3.2965973551273184, + "mean": 43.07817077636719, + "std": 1.0209636688232422 + }, + "21": { + "k_value": 3.216390969514846, + "mean": 29.16690444946289, + "std": 0.693205714225769 + }, + "22": { + "k_value": 3.313869090080252, + "mean": 27.752483367919922, + "std": 0.7432446479797363 + }, + "23": { + "k_value": 3.3482809376716407, + "mean": 66.95468139648438, + "std": 1.5324827432632446 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station21.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station21.json new file mode 100644 index 0000000..1697088 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station21.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 21, + "run_range": "260371 - 260484", + "excluded_runs": null, + "n_events": 78421, + "trigger_type": "FORCE", + "start_time": "2026-04-20T07:04:32", + "end_time": "2026-04-30T01:35:28", + "comment": "k values above 5 are set to 4: {'Ch18': 5.9112193775177015, 'Ch19': 9.505084095001225, 'Ch20': 6.588159933090225, 'Ch23': 5.194394607543958}\nk values below 3 are set to 3: {'Ch0': 1.4422257971763643, 'Ch2': 2.405412578582783, 'Ch5': 1.474207413196567, 'Ch6': 1.1416421580314684, 'Ch7': 0.9297322010993994, 'Ch9': 1.8443362140655635, 'Ch10': 2.894792032241822, 'Ch17': 2.3586874246597307, 'Ch22': 1.7476321840286293}" + }, + "values": { + "0": { + "k_value": 3.0, + "mean": 7.152980327606201, + "std": 0.4653056561946869 + }, + "1": { + "k_value": 3.457287635803239, + "mean": 7.209461688995361, + "std": 0.22343774139881134 + }, + "2": { + "k_value": 3.0, + "mean": 7.3544602394104, + "std": 0.2684972286224365 + }, + "3": { + "k_value": 3.2504649829864642, + "mean": 4.485315322875977, + "std": 0.21563079953193665 + }, + "4": { + "k_value": 4.651503105163576, + "mean": 4.612489223480225, + "std": 0.17956578731536865 + }, + "5": { + "k_value": 3.0, + "mean": 11.998124122619629, + "std": 0.730580747127533 + }, + "6": { + "k_value": 3.0, + "mean": 12.251123428344727, + "std": 0.9608492255210876 + }, + "7": { + "k_value": 3.0, + "mean": 9.393624305725098, + "std": 0.9343860149383545 + }, + "8": { + "k_value": 4.841464414596578, + "mean": 7.02142858505249, + "std": 0.2182530164718628 + }, + "9": { + "k_value": 3.0, + "mean": 10.975899696350098, + "std": 0.533765971660614 + }, + "10": { + "k_value": 3.0, + "mean": 9.08276081085205, + "std": 0.27350476384162903 + }, + "11": { + "k_value": 3.340900530815129, + "mean": 11.278002738952637, + "std": 0.2960226833820343 + }, + "12": { + "k_value": 4.567229280471803, + "mean": 8.850225448608398, + "std": 0.8977561593055725 + }, + "13": { + "k_value": 3.4933649921417245, + "mean": 8.485532760620117, + "std": 1.3222293853759766 + }, + "14": { + "k_value": 4.9689221096038905, + "mean": 9.109102249145508, + "std": 0.8420288562774658 + }, + "15": { + "k_value": 4.370627698898316, + "mean": 9.096708297729492, + "std": 1.2817699909210205 + }, + "16": { + "k_value": 4.879337778091437, + "mean": 10.197808265686035, + "std": 4.4961256980896 + }, + "17": { + "k_value": 3.0, + "mean": 8.568511962890625, + "std": 0.922333300113678 + }, + "18": { + "k_value": 4.0, + "mean": 9.374066352844238, + "std": 1.5731698274612427 + }, + "19": { + "k_value": 4.0, + "mean": 11.059850692749023, + "std": 6.396977424621582 + }, + "20": { + "k_value": 4.0, + "mean": 10.111403465270996, + "std": 0.8409211039543152 + }, + "21": { + "k_value": 3.4567768430710157, + "mean": 9.225532531738281, + "std": 0.24256783723831177 + }, + "22": { + "k_value": 3.0, + "mean": 8.453298568725586, + "std": 0.4296704828739166 + }, + "23": { + "k_value": 4.0, + "mean": 2.829272747039795, + "std": 0.26965728402137756 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station22.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station22.json new file mode 100644 index 0000000..bac45d3 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station22.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 22, + "run_range": "260049 - 260163", + "excluded_runs": null, + "n_events": 69135, + "trigger_type": "FORCE", + "start_time": "2026-04-19T23:41:29", + "end_time": "2026-04-30T01:13:08", + "comment": "k values above 5 are set to 4: {'Ch11': 6.252807588577176, 'Ch12': 5.610259827613645, 'Ch13': 12.995452182769563, 'Ch14': 5.2711795015332354, 'Ch16': 11.012882978439311, 'Ch19': 13.264011859893229, 'Ch20': 6.944240890502708}" + }, + "values": { + "0": { + "k_value": 3.556812683105463, + "mean": 5.154608249664307, + "std": 0.14652082324028015 + }, + "1": { + "k_value": 3.296257189273811, + "mean": 9.056065559387207, + "std": 0.26502251625061035 + }, + "2": { + "k_value": 3.538854228019695, + "mean": 8.347243309020996, + "std": 0.2205783873796463 + }, + "3": { + "k_value": 3.9724775533675953, + "mean": 6.360457420349121, + "std": 0.1804622858762741 + }, + "4": { + "k_value": 3.6538238439559607, + "mean": 7.110218524932861, + "std": 0.18634814023971558 + }, + "5": { + "k_value": 3.403018505096427, + "mean": 11.374512672424316, + "std": 0.3035060465335846 + }, + "6": { + "k_value": 3.3681276545524543, + "mean": 13.129602432250977, + "std": 0.3465861976146698 + }, + "7": { + "k_value": 3.4665404667853768, + "mean": 12.7036714553833, + "std": 0.35996851325035095 + }, + "8": { + "k_value": 3.436528157234191, + "mean": 8.317628860473633, + "std": 0.21785655617713928 + }, + "9": { + "k_value": 3.4398229994773732, + "mean": 8.618826866149902, + "std": 0.22164417803287506 + }, + "10": { + "k_value": 3.2880501475333417, + "mean": 4.934725761413574, + "std": 0.1341889351606369 + }, + "11": { + "k_value": 4.0, + "mean": 4.863097667694092, + "std": 0.15211918950080872 + }, + "12": { + "k_value": 4.0, + "mean": 14.215849876403809, + "std": 0.7279939651489258 + }, + "13": { + "k_value": 4.0, + "mean": 12.358235359191895, + "std": 1.6241580247879028 + }, + "14": { + "k_value": 4.0, + "mean": 14.024286270141602, + "std": 0.5120418667793274 + }, + "15": { + "k_value": 4.2398152999875505, + "mean": 14.935513496398926, + "std": 0.4175463914871216 + }, + "16": { + "k_value": 4.0, + "mean": 13.659889221191406, + "std": 0.7517056465148926 + }, + "17": { + "k_value": 4.3576132974622315, + "mean": 13.930728912353516, + "std": 0.4480227530002594 + }, + "18": { + "k_value": 4.807732987403654, + "mean": 14.666962623596191, + "std": 0.46979859471321106 + }, + "19": { + "k_value": 4.0, + "mean": 13.411582946777344, + "std": 1.2750240564346313 + }, + "20": { + "k_value": 4.0, + "mean": 14.861200332641602, + "std": 0.5816711187362671 + }, + "21": { + "k_value": 3.2966446547508013, + "mean": 11.672904968261719, + "std": 0.3117755353450775 + }, + "22": { + "k_value": 3.2906630415916185, + "mean": 9.873927116394043, + "std": 0.25367113947868347 + }, + "23": { + "k_value": 3.823026528358448, + "mean": 10.052628517150879, + "std": 0.27468645572662354 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station23.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station23.json new file mode 100644 index 0000000..0e3c9f4 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station23.json @@ -0,0 +1,142 @@ +{ + "metadata": { + "station_id": 23, + "run_range": "260192 - 260238", + "excluded_runs": [ + 260197, + 260199, + 260200, + 260201, + 260205, + 260208, + 260211 + ], + "n_events": 20270, + "trigger_type": "FORCE", + "start_time": "2026-04-25T23:23:11", + "end_time": "2026-04-30T00:13:46", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.194217198371886, + "mean": 21.56922149658203, + "std": 0.5370182394981384 + }, + "1": { + "k_value": 3.3317367660999264, + "mean": 30.919767379760742, + "std": 0.8519793152809143 + }, + "2": { + "k_value": 3.403001286029815, + "mean": 35.02701187133789, + "std": 0.8586699366569519 + }, + "3": { + "k_value": 3.247984070539474, + "mean": 20.834453582763672, + "std": 0.7508338689804077 + }, + "4": { + "k_value": 3.362138323545456, + "mean": 34.53500747680664, + "std": 0.8284710049629211 + }, + "5": { + "k_value": 3.275344269275665, + "mean": 16.435007095336914, + "std": 0.4191322326660156 + }, + "6": { + "k_value": 3.328954918146125, + "mean": 22.321317672729492, + "std": 0.569872260093689 + }, + "7": { + "k_value": 3.324644407749172, + "mean": 14.733354568481445, + "std": 0.3739366829395294 + }, + "8": { + "k_value": 3.23531752371788, + "mean": 33.62776184082031, + "std": 0.8047714829444885 + }, + "9": { + "k_value": 3.245670686721801, + "mean": 39.97263717651367, + "std": 0.975435733795166 + }, + "10": { + "k_value": 3.2590417263507767, + "mean": 41.459712982177734, + "std": 1.0297974348068237 + }, + "11": { + "k_value": 3.2226814892292, + "mean": 29.693925857543945, + "std": 0.7136784791946411 + }, + "12": { + "k_value": 3.2728459746837544, + "mean": 14.883674621582031, + "std": 0.3794845640659332 + }, + "13": { + "k_value": 3.1461129264831507, + "mean": 12.291208267211914, + "std": 0.39771682024002075 + }, + "14": { + "k_value": 3.2775473947525025, + "mean": 14.667354583740234, + "std": 0.3663754165172577 + }, + "15": { + "k_value": 3.280735691785812, + "mean": 15.26251220703125, + "std": 0.3861560523509979 + }, + "16": { + "k_value": 3.527086515426631, + "mean": 13.305425643920898, + "std": 0.36312028765678406 + }, + "17": { + "k_value": 3.350110022068022, + "mean": 15.021873474121094, + "std": 0.37982791662216187 + }, + "18": { + "k_value": 3.3344108855724297, + "mean": 14.780406951904297, + "std": 0.3764530122280121 + }, + "19": { + "k_value": 3.3265269834995195, + "mean": 15.53180980682373, + "std": 0.5026963353157043 + }, + "20": { + "k_value": 3.1421076207160907, + "mean": 15.394103050231934, + "std": 0.40808844566345215 + }, + "21": { + "k_value": 3.262706562280649, + "mean": 25.77083396911621, + "std": 0.6317614316940308 + }, + "22": { + "k_value": 3.286234536409375, + "mean": 43.81574249267578, + "std": 1.0536906719207764 + }, + "23": { + "k_value": 3.4295226328372856, + "mean": 29.83637809753418, + "std": 0.7495086789131165 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station24.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station24.json new file mode 100644 index 0000000..786cb10 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station24.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 24, + "run_range": "260000 - 260054", + "excluded_runs": null, + "n_events": 36685, + "trigger_type": "FORCE", + "start_time": "2026-04-25T03:42:06", + "end_time": "2026-04-30T00:20:22", + "comment": "k values above 5 are set to 4: {'Ch0': 6.025253431320186, 'Ch2': 7.225920055389383, 'Ch3': 7.350189552307124, 'Ch5': 7.900890697479242, 'Ch7': 6.088271541595452, 'Ch10': 7.281723619461008, 'Ch21': 6.630973987579344}\nk values below 3 are set to 3: {'Ch11': 2.8676452941894466, 'Ch13': 2.8297007513046246}" + }, + "values": { + "0": { + "k_value": 4.0, + "mean": 8.643484115600586, + "std": 0.2876773476600647 + }, + "1": { + "k_value": 4.15399373245239, + "mean": 11.779861450195312, + "std": 0.3485592007637024 + }, + "2": { + "k_value": 4.0, + "mean": 7.30964469909668, + "std": 0.29489463567733765 + }, + "3": { + "k_value": 4.0, + "mean": 8.770387649536133, + "std": 0.34533342719078064 + }, + "4": { + "k_value": 4.111735189437811, + "mean": 19.551294326782227, + "std": 0.5630038380622864 + }, + "5": { + "k_value": 4.0, + "mean": 5.657017707824707, + "std": 0.38930246233940125 + }, + "6": { + "k_value": 3.4641116456985426, + "mean": 12.349453926086426, + "std": 0.4487558901309967 + }, + "7": { + "k_value": 4.0, + "mean": 10.520540237426758, + "std": 0.3209836781024933 + }, + "8": { + "k_value": 3.768948442459097, + "mean": 22.2974910736084, + "std": 0.6374647617340088 + }, + "9": { + "k_value": 4.763645530700675, + "mean": 12.802082061767578, + "std": 0.3899436593055725 + }, + "10": { + "k_value": 4.0, + "mean": 8.004475593566895, + "std": 0.31474873423576355 + }, + "11": { + "k_value": 3.0, + "mean": 19.49131202697754, + "std": 0.8335302472114563 + }, + "12": { + "k_value": 3.6290010671615534, + "mean": 19.52635383605957, + "std": 0.6390838623046875 + }, + "13": { + "k_value": 3.0, + "mean": 16.689565658569336, + "std": 0.7082719206809998 + }, + "14": { + "k_value": 3.3189828786849964, + "mean": 17.863292694091797, + "std": 0.5975767970085144 + }, + "15": { + "k_value": 3.7834213819503613, + "mean": 17.890628814697266, + "std": 0.550787627696991 + }, + "16": { + "k_value": 3.386191650390602, + "mean": 17.888456344604492, + "std": 0.5733327865600586 + }, + "17": { + "k_value": 4.058587656021118, + "mean": 18.69788360595703, + "std": 0.5588493347167969 + }, + "18": { + "k_value": 3.6604526367187247, + "mean": 17.44533348083496, + "std": 0.5308471918106079 + }, + "19": { + "k_value": 3.897598328590381, + "mean": 16.89708137512207, + "std": 0.5523017048835754 + }, + "20": { + "k_value": 4.358631706237764, + "mean": 16.05838966369629, + "std": 0.479856014251709 + }, + "21": { + "k_value": 4.0, + "mean": 10.168705940246582, + "std": 0.3807085454463959 + }, + "22": { + "k_value": 4.541613552093504, + "mean": 14.570469856262207, + "std": 0.4116036891937256 + }, + "23": { + "k_value": 4.667956142425488, + "mean": 13.039680480957031, + "std": 0.5283921957015991 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station25.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station25.json new file mode 100644 index 0000000..38c34b5 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms/expected_rms_station25.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 25, + "run_range": "26136 - 26141", + "excluded_runs": null, + "n_events": 4238, + "trigger_type": "FORCE", + "start_time": "2026-08-04T21:52:54", + "end_time": "2026-08-05T09:52:43", + "comment": "k values above 5 are set to 4: {'Ch15': 5.1442896857261475, 'Ch16': 5.128233953952767, 'Ch17': 6.438942152976932, 'Ch18': 5.870298740863792, 'Ch19': 6.747123664855915}" + }, + "values": { + "0": { + "k_value": 3.360272208929061, + "mean": 7.029661178588867, + "std": 0.1946384608745575 + }, + "1": { + "k_value": 3.2861254136562335, + "mean": 1.2389384508132935, + "std": 0.032614849507808685 + }, + "2": { + "k_value": 3.032745650529858, + "mean": 4.5214338302612305, + "std": 0.1232999935746193 + }, + "3": { + "k_value": 3.300500049829472, + "mean": 4.43020486831665, + "std": 0.11940082907676697 + }, + "4": { + "k_value": 3.1147743377685524, + "mean": 2.609440326690674, + "std": 0.06970223784446716 + }, + "5": { + "k_value": 3.2004999961852927, + "mean": 2.9325387477874756, + "std": 0.07592901587486267 + }, + "6": { + "k_value": 3.114688693523407, + "mean": 2.3421032428741455, + "std": 0.06301853060722351 + }, + "7": { + "k_value": 3.2488546133041334, + "mean": 4.265766143798828, + "std": 0.1159258708357811 + }, + "8": { + "k_value": 3.2697022154331075, + "mean": 4.254520893096924, + "std": 0.11404559016227722 + }, + "9": { + "k_value": 3.1824481041431283, + "mean": 4.134167194366455, + "std": 0.11073244363069534 + }, + "10": { + "k_value": 3.2971972119808193, + "mean": 5.34270715713501, + "std": 0.14318329095840454 + }, + "11": { + "k_value": 3.0674549961090056, + "mean": 6.026305198669434, + "std": 0.16511060297489166 + }, + "12": { + "k_value": 4.318484922409032, + "mean": 2.2214672565460205, + "std": 0.06709767878055573 + }, + "13": { + "k_value": 4.214860142707816, + "mean": 2.1569159030914307, + "std": 0.06319107115268707 + }, + "14": { + "k_value": 4.694547227382604, + "mean": 2.2115509510040283, + "std": 0.06744652986526489 + }, + "15": { + "k_value": 4.0, + "mean": 2.132081985473633, + "std": 0.06508462876081467 + }, + "16": { + "k_value": 4.0, + "mean": 2.450115442276001, + "std": 0.0849505364894867 + }, + "17": { + "k_value": 4.0, + "mean": 2.312035083770752, + "std": 0.08146201819181442 + }, + "18": { + "k_value": 4.0, + "mean": 2.351181983947754, + "std": 0.08378539979457855 + }, + "19": { + "k_value": 4.0, + "mean": 2.2806851863861084, + "std": 0.08067978918552399 + }, + "20": { + "k_value": 3.4665372905730965, + "mean": 4.322780132293701, + "std": 0.11613284051418304 + }, + "21": { + "k_value": 3.3500478758811885, + "mean": 3.21541428565979, + "std": 0.08966273814439774 + }, + "22": { + "k_value": 3.428150151729574, + "mean": 3.7888944149017334, + "std": 0.10406816005706787 + }, + "23": { + "k_value": 3.137301130294799, + "mean": 2.8291118144989014, + "std": 0.07869267463684082 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms_values.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms_values.py new file mode 100644 index 0000000..09574f5 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_rms_values.py @@ -0,0 +1,234 @@ +'''This module can be used to find expected RMS (or Vrms) values for RNO-G stations from a known stable time period.''' +import logging +import os +import numpy as np +from argparse import ArgumentParser +import json +import sys +import pandas as pd + +SCRIPT_DIR_REF = os.path.dirname(os.path.abspath(__file__)) + +EXPECTED_VALUES_DIR_REF = os.path.join(SCRIPT_DIR_REF, f"expected_rms") +PLOTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "plots_reference/rms") +LOGS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "logs_reference/rms") +RESULTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "results_reference/rms") + +os.makedirs(PLOTS_DIR_REF, exist_ok=True) +os.makedirs(LOGS_DIR_REF, exist_ok=True) +os.makedirs(RESULTS_DIR_REF, exist_ok=True) +os.makedirs(EXPECTED_VALUES_DIR_REF, exist_ok=True) + +PARENT_DIR = os.path.dirname(SCRIPT_DIR_REF) +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") +sys.path.insert(0, PARENT_DIR) + +logger = logging.getLogger(__name__) + +from analysis_functions_sva.z_score_analysis_sva import outlier_details, calculate_z_score_parameter, find_k_value, save_values_json, outlier_flag, find_outlier_details, calculate_z_score_rolling, metadata_dict, calculate_expected_values_per_trigger +from plotting_functions_sva.plotting_sva_vrms import plot_vrms_values_against_time_single_trigger_zscore, plot_rolling_mean_std, plot_rolling_mean_linregress, create_heatmap_plot +from monitoring_data_functions_sva.get_monitoring_data_uproot import read_multiple_runs, choose_trigger_type_header +from analysis_functions_sva.vrms_analysis_sva import get_rms_per_trigger_monitoring, calculate_vrms +from analysis_functions_sva.vrms_stability_analysis_sva import get_rms_per_run, relative_median_shift, decision_metric +from helper_functions.read_rnog_runtable import read_rnog_runtable +from helper_functions.output_writer import write_failed_runs_to_csv, write_vrms_outlier_details +from helper_functions.config_helper import get_station_config + +def setup_logging(station_id, run_label): + + log_file = os.path.join(LOGS_DIR_REF, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis - Expected RMS values.") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + argparser.add_argument("--data_location", type=str, default="desy", help="Location of the data. Use 'desy' (inbox data), 'uchicago' (mirrored data) or provide a custom path to the data directory, e.g. --data_location /path/to/data") + argparser.add_argument("--save-values", action="store_true", help="Whether to save the calculated reference values as JSON files in the script directory, e.g. --save-values") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + args = argparser.parse_args() + use_monitoring = True + + parameter_label = "rms" + + station_id = args.station_id + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + setup_logging(station_id, run_label) + + logger.info("Using monitoring method to read data") + + # Choose the data location based on the argument provided and define the save location for the results + if args.data_location == "desy": + logger.info("Using DESY inbox data location for the analysis.") + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + + elif args.data_location == "uchicago": + logger.info("Using UChicago mirrored data location for the analysis.") + base_data_path = "/data/satellite" + + else: + logger.info(f"Using custom data location {args.data_location} for the analysis.") + base_data_path = args.data_location + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + all_channels = config["all_channels"] + + # Choose RADIANT or DIDAQ based on the station configuration + digitizer_type = config["daq_type"] + if digitizer_type not in ["radiant", "didaq"]: + logger.error(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'.") + raise ValueError(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'. Please check the station configuration in config_station.json.") + + if digitizer_type == "radiant": + trigger_types_daq = {"force": "FORCE", "lt": "LT", "radiant0": "RADIANT0", "radiant1": "RADIANT1"} + elif digitizer_type == "didaq": + trigger_types_daq = {"force": "FORCE", "lt": "DIDAQ_DEEP_PHASED", "radiant0": "DIDAQ_SURF_UP", "radiant1": "DIDAQ_SURF_DOWN"} + + combined_event_info = read_multiple_runs(base_path = base_data_path, station_id = station_id, run_numbers=run_numbers, daq_type=digitizer_type) + rms_arr = combined_event_info["rms_arr"] + trigger_type_arr = combined_event_info["triggerType"] + times = combined_event_info["trigger_time_utc"] + run_no = combined_event_info["run_no"] + event_number_arr = combined_event_info["event_number_arr"] + failed_run_info = combined_event_info["failed_run_info"] or {} + failed_runs = list(failed_run_info.keys()) + + valid_times_mask = ~pd.isna(times) + if np.any(~valid_times_mask): + invalid_runs = np.unique(run_no[~valid_times_mask]) + logger.warning(f"Found {np.sum(~valid_times_mask)} invalid timestamps in runs {invalid_runs}. These events will be skipped in the analysis.") + times = times[valid_times_mask] + rms_arr = rms_arr[:, valid_times_mask] + trigger_type_arr = trigger_type_arr[valid_times_mask] + run_no = run_no[valid_times_mask] + event_number_arr = event_number_arr[valid_times_mask] + + for invalid_run in invalid_runs: + failed_run_info[invalid_run] = "Some events have been skipped in the analysis due to invalid timestamps, check logs for details" + + force_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["force"], digitizer_type) + lt_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["lt"], digitizer_type) + radiant0_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["radiant0"], digitizer_type) + radiant1_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["radiant1"], digitizer_type) + + run_no_force = run_no[force_mask] + event_number_force = event_number_arr[force_mask] + + run_no_radiant0 = run_no[radiant0_mask] + event_number_radiant0 = event_number_arr[radiant0_mask] + + run_no_radiant1 = run_no[radiant1_mask] + event_number_radiant1 = event_number_arr[radiant1_mask] + + run_no_lt = run_no[lt_mask] + event_number_lt = event_number_arr[lt_mask] + + vrms_arr,vrms_arr_force, vrms_arr_radiant0, vrms_arr_radiant1, vrms_arr_lt = get_rms_per_trigger_monitoring(rms_arr=rms_arr, force_mask=force_mask, lt_mask=lt_mask, radiant0_mask=radiant0_mask, radiant1_mask=radiant1_mask) + + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=RESULTS_DIR_REF) + + logger.info(f"Start calculating expected values for {parameter_label} parameter...") + times_force = times[force_mask] + times_radiant0 = times[radiant0_mask] + times_radiant1 = times[radiant1_mask] + times_lt = times[lt_mask] + + z_score_force, z_score_rolling_force, k_values_force, vrms_mean_force, vrms_std_force, metadata_force, rolling_mean_force, rolling_std_force = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_force, times_force, trigger_type=trigger_types_daq["force"], excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_radiant0, z_score_rolling_radiant0, k_values_radiant0, vrms_mean_radiant0, vrms_std_radiant0, metadata_radiant0, rolling_mean_radiant0, rolling_std_radiant0 = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_radiant0, times_radiant0, trigger_type=trigger_types_daq["radiant0"], excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_radiant1, z_score_rolling_radiant1, k_values_radiant1, vrms_mean_radiant1, vrms_std_radiant1, metadata_radiant1, rolling_mean_radiant1, rolling_std_radiant1 = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_radiant1, times_radiant1, trigger_type=trigger_types_daq["radiant1"], excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_lt, z_score_rolling_lt, k_values_lt, vrms_mean_lt, vrms_std_lt, metadata_lt, rolling_mean_lt, rolling_std_lt = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_lt, times_lt, trigger_type=trigger_types_daq["lt"], excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + + if args.save_values: + save_values_json(k_values_force, vrms_mean_force, vrms_std_force, filename=f"expected_{parameter_label}_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_force) + # save_values_json(k_values_radiant0, vrms_mean_radiant0, vrms_std_radiant0, filename=f"expected_{parameter_label}_radiant0_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_radiant0) + # save_values_json(k_values_radiant1, vrms_mean_radiant1, vrms_std_radiant1, filename=f"expected_{parameter_label}_radiant1_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_radiant1) + # save_values_json(k_values_lt, vrms_mean_lt, vrms_std_lt, filename=f"expected_{parameter_label}_lt_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_lt) + + flag_outliers_force, outlier_details_force = outlier_details(z_score_force, k_values_force, all_channels, run_no_force, event_number_force, trigger_label=trigger_types_daq["force"]) + flag_outliers_radiant0, outlier_details_radiant0 = outlier_details(z_score_radiant0, k_values_radiant0, all_channels, run_no_radiant0, event_number_radiant0, trigger_label=trigger_types_daq["radiant0"]) + flag_outliers_radiant1, outlier_details_radiant1 = outlier_details(z_score_radiant1, k_values_radiant1, all_channels, run_no_radiant1, event_number_radiant1, trigger_label=trigger_types_daq["radiant1"]) + flag_outliers_lt, outlier_details_lt = outlier_details(z_score_lt, k_values_lt, all_channels, run_no_lt, event_number_lt, trigger_label=trigger_types_daq["lt"]) + + k_values_rolling = {int(ch): 4 for ch in all_channels} # Placeholder, as k-values for rolling z-score are not calculated in this script, but could be implemented in the future if needed + flag_outliers_force_rolling, outlier_details_force_rolling = outlier_details(z_score_rolling_force, k_values_rolling, all_channels, run_no_force, event_number_force, trigger_label=f"{trigger_types_daq['force']}_rolling") + + write_vrms_outlier_details(outlier_details_force, station_id, run_label, trigger_label=trigger_types_daq["force"], n_events = len(times_force), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_radiant0, station_id, run_label, trigger_label=trigger_types_daq["radiant0"], n_events = len(times_radiant0), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_radiant1, station_id, run_label, trigger_label=trigger_types_daq["radiant1"], n_events = len(times_radiant1), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_lt, station_id, run_label, trigger_label=trigger_types_daq["lt"], n_events = len(times_lt), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_force_rolling, station_id, run_label, trigger_label=f"{trigger_types_daq['force']}_rolling", n_events = len(times_force), results_dir=RESULTS_DIR_REF) + + plot_vrms_values_against_time_single_trigger_zscore(times_force, vrms_arr_force, flag_outliers_force, z_score_force, k_values_force, trigger_name=trigger_types_daq["force"], channel_list=all_channels, station_id=station_id, run_label=run_label, save_location=PLOTS_DIR_REF, use_monitoring=True) + rms_arr_per_run_dict_force = get_rms_per_run(vrms_arr_force, run_no_force) + + relative_median_shift_results = relative_median_shift(rms_arr_per_run_dict_force, all_channels) + + with open(os.path.join(RESULTS_DIR_REF, f"rms_relative_median_shift_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(relative_median_shift_results, f, indent=4) + + create_heatmap_plot(relative_median_shift_results, label = "Relative Median Shift", save_dir = PLOTS_DIR_REF, channel_list=all_channels, station_id = station_id,matrix_key = "median_shift_matrix", run_label=run_label, cmap="Reds") + + rms_results = decision_metric(outlier_details_force, relative_median_shift_results, n_events_force=len(times_force), channels=all_channels) + with open(os.path.join(RESULTS_DIR_REF, f"rms_stability_decision_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(rms_results, f, indent=4) diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station11.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station11.json new file mode 100644 index 0000000..6de4ce5 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station11.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 11, + "run_range": "260532 - 260649", + "excluded_runs": null, + "n_events": 76288, + "trigger_type": "FORCE", + "start_time": "NaT", + "end_time": "NaT", + "comment": "k values above 5 are set to 4: {'Ch13': 5.1577884815725845}" + }, + "values": { + "0": { + "k_value": 3.7686170126140417, + "mean": 0.5845983089878136, + "std": 0.03849581941673669 + }, + "1": { + "k_value": 3.7202923318277183, + "mean": 0.5720006486050042, + "std": 0.037149949564306436 + }, + "2": { + "k_value": 3.697085598528713, + "mean": 0.5953244427253698, + "std": 0.04111994305746344 + }, + "3": { + "k_value": 3.7391211261078037, + "mean": 0.6063770132841899, + "std": 0.042377474303430196 + }, + "4": { + "k_value": 3.828555384987946, + "mean": 0.5582302248204978, + "std": 0.03630893981122643 + }, + "5": { + "k_value": 3.8504504215703164, + "mean": 0.5596333454791592, + "std": 0.03642854920842437 + }, + "6": { + "k_value": 3.786532011316184, + "mean": 0.5599290245570983, + "std": 0.036443870675409135 + }, + "7": { + "k_value": 4.144195081025989, + "mean": 0.5819441475380119, + "std": 0.03890040708029448 + }, + "8": { + "k_value": 3.754966549968525, + "mean": 0.5709098722617276, + "std": 0.0367228689088193 + }, + "9": { + "k_value": 3.8064368370037585, + "mean": 0.566442088156092, + "std": 0.0369315535690418 + }, + "10": { + "k_value": 3.758384580394858, + "mean": 0.5634714317615644, + "std": 0.036750889271517574 + }, + "11": { + "k_value": 3.7597057619916163, + "mean": 0.5708883218665877, + "std": 0.03682288296949154 + }, + "12": { + "k_value": 3.830037799468466, + "mean": 0.617579334384736, + "std": 0.05225793399154064 + }, + "13": { + "k_value": 4.0, + "mean": 0.595756057005961, + "std": 0.04232910254640879 + }, + "14": { + "k_value": 4.167863793304793, + "mean": 0.591360003930067, + "std": 0.04004372645886919 + }, + "15": { + "k_value": 3.6527639094142956, + "mean": 0.5944561212568933, + "std": 0.039824900849087956 + }, + "16": { + "k_value": 4.265287696379271, + "mean": 0.6025825293490901, + "std": 0.04176956822124566 + }, + "17": { + "k_value": 3.757956629870979, + "mean": 0.59645926543001, + "std": 0.040033376547531196 + }, + "18": { + "k_value": 3.8581550575523833, + "mean": 0.598629036396023, + "std": 0.04061477010566624 + }, + "19": { + "k_value": 4.349435949792409, + "mean": 0.5984452039541481, + "std": 0.04123806859450991 + }, + "20": { + "k_value": 3.7442759518797413, + "mean": 0.5973081253621534, + "std": 0.03994043039579265 + }, + "21": { + "k_value": 3.6868018558680946, + "mean": 0.5859365974257773, + "std": 0.03836904621247123 + }, + "22": { + "k_value": 3.629478074424344, + "mean": 0.5985345226160163, + "std": 0.040815125699995125 + }, + "23": { + "k_value": 3.727554193840063, + "mean": 0.5775995750005583, + "std": 0.03807787201926971 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station12.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station12.json new file mode 100644 index 0000000..cd99959 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station12.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 12, + "run_range": "260425 - 260518", + "excluded_runs": null, + "n_events": 25745, + "trigger_type": "FORCE", + "start_time": "2026-04-19T23:28:21", + "end_time": "2026-04-29T14:32:24", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.706658195792406, + "mean": 0.5679117460748705, + "std": 0.03687325554430942 + }, + "1": { + "k_value": 3.7951735853972974, + "mean": 0.5669057512876258, + "std": 0.03657279256669446 + }, + "2": { + "k_value": 3.6718642234992274, + "mean": 0.5977764167549173, + "std": 0.040351149864482495 + }, + "3": { + "k_value": 3.7555926592949334, + "mean": 0.5687009682968791, + "std": 0.036535659178611415 + }, + "4": { + "k_value": 3.7740209113833285, + "mean": 0.5556606023799998, + "std": 0.03629296103285548 + }, + "5": { + "k_value": 3.8772786378730433, + "mean": 0.5588766271708083, + "std": 0.036487446987463285 + }, + "6": { + "k_value": 3.8439945137322447, + "mean": 0.5610606260693244, + "std": 0.03611404857419937 + }, + "7": { + "k_value": 3.852527166553172, + "mean": 0.5615340254129628, + "std": 0.03664429813669989 + }, + "8": { + "k_value": 3.8752459172967537, + "mean": 0.5534218835952148, + "std": 0.03611744078222124 + }, + "9": { + "k_value": 3.694322178474811, + "mean": 0.5750031826323166, + "std": 0.03727275413063764 + }, + "10": { + "k_value": 3.742781609899743, + "mean": 0.5483469887677616, + "std": 0.03536614483191605 + }, + "11": { + "k_value": 3.8252769245186533, + "mean": 0.5607354990365064, + "std": 0.03621235208485301 + }, + "12": { + "k_value": 3.741034942930655, + "mean": 0.5943415633569958, + "std": 0.0396526015336902 + }, + "13": { + "k_value": 3.7312506360417115, + "mean": 0.6000110635603815, + "std": 0.040589559503327544 + }, + "14": { + "k_value": 3.710864242871587, + "mean": 0.5918240841753574, + "std": 0.039425868625951786 + }, + "15": { + "k_value": 3.766690118474879, + "mean": 0.596321825139822, + "std": 0.039893050215152576 + }, + "16": { + "k_value": 3.704674810405491, + "mean": 0.5928411891090006, + "std": 0.03962955777015182 + }, + "17": { + "k_value": 3.6212451061720126, + "mean": 0.5979459107052698, + "std": 0.03999529619801604 + }, + "18": { + "k_value": 3.7025025699363896, + "mean": 0.5903596289555887, + "std": 0.03980758347465428 + }, + "19": { + "k_value": 3.566974543979074, + "mean": 0.6005301133057516, + "std": 0.04062086501424552 + }, + "20": { + "k_value": 3.6261872651259948, + "mean": 0.6021992790979671, + "std": 0.040245707066497736 + }, + "21": { + "k_value": 3.891934438282377, + "mean": 0.5620225121412699, + "std": 0.03633904879228767 + }, + "22": { + "k_value": 3.363809808162379, + "mean": 0.5861932423277273, + "std": 0.044691936223216606 + }, + "23": { + "k_value": 3.8125925724742484, + "mean": 0.5566979315715405, + "std": 0.03629445867619719 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station13.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station13.json new file mode 100644 index 0000000..8a9e414 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station13.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 13, + "run_range": "260273 - 260383", + "excluded_runs": null, + "n_events": 62608, + "trigger_type": "FORCE", + "start_time": "2026-04-20T07:19:48", + "end_time": "2026-04-30T00:06:26", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.8530302394202343, + "mean": 0.6199884266638325, + "std": 0.04896551903291796 + }, + "1": { + "k_value": 3.6209479093619934, + "mean": 0.5924643937950012, + "std": 0.03882170596307034 + }, + "2": { + "k_value": 3.777698358252876, + "mean": 0.6015332668713093, + "std": 0.04158814360678907 + }, + "3": { + "k_value": 3.6800693597740497, + "mean": 0.5817592002108375, + "std": 0.03798030544116006 + }, + "4": { + "k_value": 3.7923116126772864, + "mean": 0.5709694540795051, + "std": 0.03686203730586055 + }, + "5": { + "k_value": 3.743605634676807, + "mean": 0.5555084745563643, + "std": 0.03599846858219632 + }, + "6": { + "k_value": 3.786015985318763, + "mean": 0.5643479576980791, + "std": 0.03645591235945147 + }, + "7": { + "k_value": 3.924314517708718, + "mean": 0.5555624251867519, + "std": 0.035912291556931174 + }, + "8": { + "k_value": 3.8062266046690802, + "mean": 0.5638423777059139, + "std": 0.0364783054132751 + }, + "9": { + "k_value": 3.803580642793851, + "mean": 0.5599779023880382, + "std": 0.03653840636218633 + }, + "10": { + "k_value": 3.8537198712691194, + "mean": 0.5695624725922319, + "std": 0.03683873401634664 + }, + "11": { + "k_value": 3.747759774829978, + "mean": 0.5737951480625472, + "std": 0.03701650633844059 + }, + "12": { + "k_value": 3.7162380924080227, + "mean": 0.5981133631541006, + "std": 0.039641444388342724 + }, + "13": { + "k_value": 3.783441520381424, + "mean": 0.5952226167242041, + "std": 0.040431697149254896 + }, + "14": { + "k_value": 3.7618843414217746, + "mean": 0.5935540001351151, + "std": 0.03940792704085979 + }, + "15": { + "k_value": 3.6960114210983486, + "mean": 0.5982819353787696, + "std": 0.04012604220280451 + }, + "16": { + "k_value": 3.633251785299148, + "mean": 0.6050811511609292, + "std": 0.041385163049107755 + }, + "17": { + "k_value": 3.778390899930184, + "mean": 0.5968252646200374, + "std": 0.039931346704769125 + }, + "18": { + "k_value": 3.7742174791115737, + "mean": 0.5922990659413804, + "std": 0.03984389556589627 + }, + "19": { + "k_value": 3.8482751268005266, + "mean": 0.6027097542624914, + "std": 0.041401883780147364 + }, + "20": { + "k_value": 3.7126189093578184, + "mean": 0.5944193757387688, + "std": 0.03949780773551082 + }, + "21": { + "k_value": 3.791006887530518, + "mean": 0.5603430369606032, + "std": 0.03607845387446088 + }, + "22": { + "k_value": 3.79812670048577, + "mean": 0.5761441439949476, + "std": 0.03736445233415921 + }, + "23": { + "k_value": 3.860131553286178, + "mean": 0.5599938394801163, + "std": 0.03668325853149469 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station14.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station14.json new file mode 100644 index 0000000..f1236c7 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station14.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 14, + "run_range": "260080 - 260150", + "excluded_runs": null, + "n_events": 50856, + "trigger_type": "FORCE", + "start_time": "2026-04-26T09:28:44", + "end_time": "2026-05-02T11:09:33", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.7560450885233387, + "mean": 0.5684536006147282, + "std": 0.03697195383632088 + }, + "1": { + "k_value": 3.7800711938738156, + "mean": 0.5984913343726836, + "std": 0.04014828819341104 + }, + "2": { + "k_value": 3.7678671801170864, + "mean": 0.5601123051212809, + "std": 0.03602426537419151 + }, + "3": { + "k_value": 3.837086795688993, + "mean": 0.5619942302523575, + "std": 0.03616210872852611 + }, + "4": { + "k_value": 3.814943042046713, + "mean": 0.5834154672372017, + "std": 0.03801574151672666 + }, + "5": { + "k_value": 3.7393445027248604, + "mean": 0.5628643323950285, + "std": 0.03650962113585922 + }, + "6": { + "k_value": 3.7785063671134993, + "mean": 0.5575570752649583, + "std": 0.03598119582998781 + }, + "7": { + "k_value": 3.8039618078735242, + "mean": 0.5584591439529778, + "std": 0.03614725328579431 + }, + "8": { + "k_value": 3.759744846039237, + "mean": 0.5517400401868175, + "std": 0.03564565060404225 + }, + "9": { + "k_value": 3.8854998058541597, + "mean": 0.5540047044736897, + "std": 0.03612316268287032 + }, + "10": { + "k_value": 3.772892667016383, + "mean": 0.546577138264185, + "std": 0.035225643261183215 + }, + "11": { + "k_value": 3.8414743351861587, + "mean": 0.5535433829666732, + "std": 0.03596859168734109 + }, + "12": { + "k_value": 3.7572887359786837, + "mean": 0.5577216095131241, + "std": 0.03618534735933632 + }, + "13": { + "k_value": 3.766368063728083, + "mean": 0.5596628675515237, + "std": 0.036939016655352895 + }, + "14": { + "k_value": 3.7443637743032476, + "mean": 0.5782345226359664, + "std": 0.037635780660010866 + }, + "15": { + "k_value": 3.8828849930589198, + "mean": 0.5654111311434286, + "std": 0.03651482199978443 + }, + "16": { + "k_value": 3.7898007832017044, + "mean": 0.5595240136050256, + "std": 0.036416409560917 + }, + "17": { + "k_value": 3.8657720686587007, + "mean": 0.5714523401266909, + "std": 0.037398353764665306 + }, + "18": { + "k_value": 3.8201907717744956, + "mean": 0.5529864031056577, + "std": 0.03628117283907995 + }, + "19": { + "k_value": 3.921833993893435, + "mean": 0.5575043804065444, + "std": 0.03632210898846267 + }, + "20": { + "k_value": 3.873615355529049, + "mean": 0.5577769919514014, + "std": 0.03612839688848345 + }, + "21": { + "k_value": 3.80789181435776, + "mean": 0.567967354587412, + "std": 0.03646108679200027 + }, + "22": { + "k_value": 3.901816647627106, + "mean": 0.5694754551020351, + "std": 0.03692869780129087 + }, + "23": { + "k_value": 3.8478306162316223, + "mean": 0.5505252051705589, + "std": 0.03550823062779063 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station21.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station21.json new file mode 100644 index 0000000..47ec770 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station21.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 21, + "run_range": "260371 - 260484", + "excluded_runs": null, + "n_events": 78422, + "trigger_type": "FORCE", + "start_time": "NaT", + "end_time": "NaT", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.7751536741664644, + "mean": 0.6209650263600756, + "std": 0.048713745790834546 + }, + "1": { + "k_value": 3.62898533784865, + "mean": 0.6144140044034891, + "std": 0.0491288126541708 + }, + "2": { + "k_value": 3.696500708148893, + "mean": 0.6238713850753842, + "std": 0.04654849454442276 + }, + "3": { + "k_value": 3.7067720201520564, + "mean": 0.6157146878145918, + "std": 0.055721688219763206 + }, + "4": { + "k_value": 3.6557052039838336, + "mean": 0.6214194814237335, + "std": 0.057082714160678 + }, + "5": { + "k_value": 3.7860646377944693, + "mean": 0.606500175411455, + "std": 0.04253151691501118 + }, + "6": { + "k_value": 3.9250439242488944, + "mean": 0.6076307950925034, + "std": 0.042843906225499044 + }, + "7": { + "k_value": 3.7634903975721734, + "mean": 0.6182208150261795, + "std": 0.044674235316179194 + }, + "8": { + "k_value": 4.052954552694124, + "mean": 0.6236694686829861, + "std": 0.053809483971697596 + }, + "9": { + "k_value": 3.709364372221402, + "mean": 0.6081795558833067, + "std": 0.04327293245823671 + }, + "10": { + "k_value": 3.6921604046858834, + "mean": 0.6137058186411386, + "std": 0.04551353407667806 + }, + "11": { + "k_value": 3.7385009998616896, + "mean": 0.6016604002378095, + "std": 0.04258751919312569 + }, + "12": { + "k_value": 3.7533981965065517, + "mean": 0.6181510505559866, + "std": 0.04674540323534384 + }, + "13": { + "k_value": 3.719664459091777, + "mean": 0.6153018082565718, + "std": 0.04732238158878216 + }, + "14": { + "k_value": 3.7691578071364678, + "mean": 0.6172622625600973, + "std": 0.04774014284363033 + }, + "15": { + "k_value": 3.8108263690125517, + "mean": 0.6115125613785061, + "std": 0.04714316565177926 + }, + "16": { + "k_value": 3.7210592199432186, + "mean": 0.6039433111819733, + "std": 0.05729677962310313 + }, + "17": { + "k_value": 3.7042731064660073, + "mean": 0.6171631741961564, + "std": 0.04717782608562412 + }, + "18": { + "k_value": 3.6472303155163823, + "mean": 0.6158064571856954, + "std": 0.048559649733733765 + }, + "19": { + "k_value": 4.4957397346407735, + "mean": 0.6035241883965614, + "std": 0.0576916622739875 + }, + "20": { + "k_value": 3.7699280701454465, + "mean": 0.6105633698099973, + "std": 0.04497907159737671 + }, + "21": { + "k_value": 3.774012960023972, + "mean": 0.6137106956886078, + "std": 0.046098657036016656 + }, + "22": { + "k_value": 3.679881996536226, + "mean": 0.6136623885404481, + "std": 0.04535357635568516 + }, + "23": { + "k_value": 3.629287587705251, + "mean": 0.6524223008407125, + "std": 0.08958676680609072 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station22.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station22.json new file mode 100644 index 0000000..a7b2630 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station22.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 22, + "run_range": "260049 - 260163", + "excluded_runs": null, + "n_events": 69170, + "trigger_type": "FORCE", + "start_time": "NaT", + "end_time": "NaT", + "comment": "k values above 5 are set to 4: {'Ch13': 7.953214184201574, 'Ch19': 6.527151068953314}" + }, + "values": { + "0": { + "k_value": 3.642509109488299, + "mean": 0.6166809640373374, + "std": 0.053002974106461015 + }, + "1": { + "k_value": 3.585131326245107, + "mean": 0.621962749414459, + "std": 0.04599223922674427 + }, + "2": { + "k_value": 3.7554896830678888, + "mean": 0.6065030712500108, + "std": 0.04665328395013488 + }, + "3": { + "k_value": 3.7142972842562894, + "mean": 0.6196067197595244, + "std": 0.05256138851815656 + }, + "4": { + "k_value": 3.5652713943654484, + "mean": 0.6185688323205769, + "std": 0.04913705386180894 + }, + "5": { + "k_value": 3.6006051053554855, + "mean": 0.6106587268805614, + "std": 0.04286061829363942 + }, + "6": { + "k_value": 3.699233054357916, + "mean": 0.6036031664598746, + "std": 0.04164433101028253 + }, + "7": { + "k_value": 3.6516767717248264, + "mean": 0.6083962631518067, + "std": 0.041871871930488753 + }, + "8": { + "k_value": 3.699909930901746, + "mean": 0.6118051769579961, + "std": 0.04610387017939205 + }, + "9": { + "k_value": 3.7158502326898053, + "mean": 0.6118168409479064, + "std": 0.04592214324158456 + }, + "10": { + "k_value": 3.6553379412040865, + "mean": 0.6154566056673775, + "std": 0.052592736008651095 + }, + "11": { + "k_value": 4.813100448606423, + "mean": 0.6112725774540445, + "std": 0.05483701183120282 + }, + "12": { + "k_value": 3.988544334456988, + "mean": 0.5997193541143379, + "std": 0.041296013244038525 + }, + "13": { + "k_value": 4.0, + "mean": 0.6038003464086756, + "std": 0.04627905946646852 + }, + "14": { + "k_value": 3.8734655124984276, + "mean": 0.5963893072755307, + "std": 0.0404971030368894 + }, + "15": { + "k_value": 3.8446691249178677, + "mean": 0.5989758851886706, + "std": 0.04017811718731204 + }, + "16": { + "k_value": 4.458029279487277, + "mean": 0.5967315025478518, + "std": 0.041877540051899496 + }, + "17": { + "k_value": 3.8927871982456534, + "mean": 0.598678481324385, + "std": 0.04067877874905758 + }, + "18": { + "k_value": 3.696481055577654, + "mean": 0.5973717089352983, + "std": 0.040134451762337375 + }, + "19": { + "k_value": 4.0, + "mean": 0.6027809054687173, + "std": 0.04417477004816027 + }, + "20": { + "k_value": 4.107545000014993, + "mean": 0.5978817022304006, + "std": 0.04110219287906177 + }, + "21": { + "k_value": 3.6669808742796968, + "mean": 0.6082247234286696, + "std": 0.042932896908133025 + }, + "22": { + "k_value": 3.667120429514417, + "mean": 0.6045811513190011, + "std": 0.04354850941345635 + }, + "23": { + "k_value": 3.7711763408743613, + "mean": 0.6027253409594101, + "std": 0.043826407961731 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station23.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station23.json new file mode 100644 index 0000000..54bfe98 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station23.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 23, + "run_range": "260183 - 260238", + "excluded_runs": null, + "n_events": 24106, + "trigger_type": "FORCE", + "start_time": "NaT", + "end_time": "NaT", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.762166218366993, + "mean": 0.577107117533766, + "std": 0.03732188024567012 + }, + "1": { + "k_value": 3.9257574117416296, + "mean": 0.5665740028432129, + "std": 0.037021496647155254 + }, + "2": { + "k_value": 3.744304334508179, + "mean": 0.563456718693811, + "std": 0.03638942040874929 + }, + "3": { + "k_value": 3.652013111637321, + "mean": 0.5836332761257299, + "std": 0.03812928862745274 + }, + "4": { + "k_value": 3.759176848435261, + "mean": 0.5609462342903889, + "std": 0.036175331025028505 + }, + "5": { + "k_value": 3.810244449079828, + "mean": 0.5895762463074197, + "std": 0.03854924803158611 + }, + "6": { + "k_value": 4.009136937632877, + "mean": 0.5799833435313436, + "std": 0.037825854054512005 + }, + "7": { + "k_value": 3.6570832652384255, + "mean": 0.5950602137045344, + "std": 0.04002414760393012 + }, + "8": { + "k_value": 3.87152111829529, + "mean": 0.5627683618884816, + "std": 0.036409936443868966 + }, + "9": { + "k_value": 3.66486431660238, + "mean": 0.559694691297866, + "std": 0.03609697211876191 + }, + "10": { + "k_value": 3.8081144980505095, + "mean": 0.5577947841576371, + "std": 0.036421423593066854 + }, + "11": { + "k_value": 3.79464797414064, + "mean": 0.5686149794880494, + "std": 0.036958683642687826 + }, + "12": { + "k_value": 3.595802364401457, + "mean": 0.5973883083478627, + "std": 0.04027146593340433 + }, + "13": { + "k_value": 3.7072670635100406, + "mean": 0.604245777795171, + "std": 0.04189171587198898 + }, + "14": { + "k_value": 3.7041703188720687, + "mean": 0.5948594322039539, + "std": 0.039591206599176934 + }, + "15": { + "k_value": 3.7249653806765552, + "mean": 0.5942000666512095, + "std": 0.039929388993859054 + }, + "16": { + "k_value": 3.6821162156075413, + "mean": 0.6004542980564976, + "std": 0.04070729218986243 + }, + "17": { + "k_value": 3.7375801861195304, + "mean": 0.595576948735462, + "std": 0.03899121709399086 + }, + "18": { + "k_value": 3.707868745308873, + "mean": 0.5983115492567542, + "std": 0.03977526449116828 + }, + "19": { + "k_value": 3.737245011798906, + "mean": 0.5987948131915328, + "std": 0.04060547476061093 + }, + "20": { + "k_value": 3.713975623276045, + "mean": 0.5959866958342891, + "std": 0.03986031875483929 + }, + "21": { + "k_value": 3.7291182404868035, + "mean": 0.5718106738919525, + "std": 0.03690971425293745 + }, + "22": { + "k_value": 3.8890316918370895, + "mean": 0.5576483336364043, + "std": 0.03640406246139907 + }, + "23": { + "k_value": 3.9085782892606775, + "mean": 0.5698817576065509, + "std": 0.037099824836676266 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station24.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station24.json new file mode 100644 index 0000000..7b38abf --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station24.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 24, + "run_range": "260000 - 260054", + "excluded_runs": null, + "n_events": 36685, + "trigger_type": "FORCE", + "start_time": "2026-04-25T03:42:06", + "end_time": "2026-04-30T00:20:22", + "comment": "" + }, + "values": { + "0": { + "k_value": 3.6910938457481506, + "mean": 0.6109414983677438, + "std": 0.04739964688727684 + }, + "1": { + "k_value": 3.6969420268138786, + "mean": 0.6094172733408446, + "std": 0.0425750432452171 + }, + "2": { + "k_value": 3.6799033910357983, + "mean": 0.625455291549411, + "std": 0.05109540913552262 + }, + "3": { + "k_value": 4.651628695967841, + "mean": 0.6175273544033892, + "std": 0.05126177461540616 + }, + "4": { + "k_value": 3.80722134030498, + "mean": 0.5869319464434154, + "std": 0.03862241349681498 + }, + "5": { + "k_value": 4.594847229743769, + "mean": 0.6237751397001178, + "std": 0.06439579286540212 + }, + "6": { + "k_value": 3.7563323459268148, + "mean": 0.6118402129698062, + "std": 0.044513520929917026 + }, + "7": { + "k_value": 3.5910713381102495, + "mean": 0.6144407377580108, + "std": 0.044178163258598496 + }, + "8": { + "k_value": 3.8218087788745447, + "mean": 0.578994507771345, + "std": 0.03750319496178738 + }, + "9": { + "k_value": 3.698795019435568, + "mean": 0.6065830374165068, + "std": 0.04211647565467223 + }, + "10": { + "k_value": 4.40654989375739, + "mean": 0.624989579535691, + "std": 0.053234639322785 + }, + "11": { + "k_value": 3.797465400057462, + "mean": 0.5894939728011243, + "std": 0.03848699530603596 + }, + "12": { + "k_value": 3.787357751478934, + "mean": 0.5867491022374463, + "std": 0.038197613212360426 + }, + "13": { + "k_value": 3.703131332870245, + "mean": 0.5898964287519673, + "std": 0.039639672533782075 + }, + "14": { + "k_value": 3.6705813958889104, + "mean": 0.5913071981167124, + "std": 0.039214994738938025 + }, + "15": { + "k_value": 3.7317863874395547, + "mean": 0.5914116705167146, + "std": 0.03879764796930251 + }, + "16": { + "k_value": 3.678214750983498, + "mean": 0.5912566698869168, + "std": 0.03909565578014779 + }, + "17": { + "k_value": 3.6456208843358686, + "mean": 0.5905810538724586, + "std": 0.039171804944261146 + }, + "18": { + "k_value": 3.820665025080839, + "mean": 0.5916018101471999, + "std": 0.03918847433067903 + }, + "19": { + "k_value": 3.67253802565597, + "mean": 0.5918744962629112, + "std": 0.03904248913950449 + }, + "20": { + "k_value": 3.817147358713246, + "mean": 0.591887943122695, + "std": 0.039116961805395235 + }, + "21": { + "k_value": 4.191011936981354, + "mean": 0.6136974819817247, + "std": 0.04709448648570748 + }, + "22": { + "k_value": 3.777442126408058, + "mean": 0.6050395142787837, + "std": 0.04153609084202302 + }, + "23": { + "k_value": 4.314633386235415, + "mean": 0.6044664717128196, + "std": 0.04340626237042939 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station25.json b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station25.json new file mode 100644 index 0000000..0a08552 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr/expected_snr_values_station25.json @@ -0,0 +1,134 @@ +{ + "metadata": { + "station_id": 25, + "run_range": "26136 - 26141", + "excluded_runs": null, + "n_events": 4238, + "trigger_type": "FORCE", + "start_time": "2026-08-04T21:52:54", + "end_time": "2026-08-05T09:52:43", + "comment": "k values above 5 are set to 4: {'Ch17': 5.676992438830617, 'Ch18': 5.123960342266063, 'Ch19': 6.249966886149198}" + }, + "values": { + "0": { + "k_value": 3.3731716144026276, + "mean": 1.3302944307800213, + "std": 0.012286659918044684 + }, + "1": { + "k_value": 3.3134412244829705, + "mean": 2.026223568027705, + "std": 0.011174361580179806 + }, + "2": { + "k_value": 3.2318039817517303, + "mean": 1.4949412261347057, + "std": 0.011624648073849558 + }, + "3": { + "k_value": 3.2480516347472834, + "mean": 1.504418387354323, + "std": 0.01162682557444245 + }, + "4": { + "k_value": 3.243317914453191, + "mean": 1.7177533472768804, + "std": 0.011304123131750535 + }, + "5": { + "k_value": 3.2339031150414423, + "mean": 1.6708908284647799, + "std": 0.010998408696162034 + }, + "6": { + "k_value": 3.257549709353486, + "mean": 1.7596394117012562, + "std": 0.011388918955354372 + }, + "7": { + "k_value": 3.5604107308241466, + "mean": 1.519459810940877, + "std": 0.011444836040319631 + }, + "8": { + "k_value": 3.171824822940258, + "mean": 1.5223911083430854, + "std": 0.011494336230627643 + }, + "9": { + "k_value": 3.0234446604489005, + "mean": 1.5339651277884316, + "std": 0.011340659238841467 + }, + "10": { + "k_value": 3.1451891187748275, + "mean": 1.431623208140126, + "std": 0.011606024602641391 + }, + "11": { + "k_value": 3.2037946656652334, + "mean": 1.3859154134793494, + "std": 0.01197082097869737 + }, + "12": { + "k_value": 3.897635854148439, + "mean": 1.783161325646251, + "std": 0.01263459753915724 + }, + "13": { + "k_value": 4.016602344035191, + "mean": 1.7954902633194538, + "std": 0.012289437789221943 + }, + "14": { + "k_value": 4.0755169249692385, + "mean": 1.7824399378041318, + "std": 0.01274831342232499 + }, + "15": { + "k_value": 4.723045084739275, + "mean": 1.7980230870100025, + "std": 0.012752675737983298 + }, + "16": { + "k_value": 4.528801495564618, + "mean": 1.744380543369869, + "std": 0.014304983302639651 + }, + "17": { + "k_value": 4.0, + "mean": 1.7666070967485592, + "std": 0.014469169653189982 + }, + "18": { + "k_value": 4.0, + "mean": 1.7578303479841701, + "std": 0.014615735490604983 + }, + "19": { + "k_value": 4.0, + "mean": 1.7704651065435115, + "std": 0.01458366735595731 + }, + "20": { + "k_value": 3.495254250291437, + "mean": 1.516755728292992, + "std": 0.011518279511113476 + }, + "21": { + "k_value": 3.3085573262631742, + "mean": 1.6337547815884317, + "std": 0.011739335602615598 + }, + "22": { + "k_value": 3.3153737604610556, + "mean": 1.5653321920707275, + "std": 0.011629166057333168 + }, + "23": { + "k_value": 3.1841295160204632, + "mean": 1.6826307690813456, + "std": 0.011648859194106723 + } + } +} \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr_values.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr_values.py new file mode 100644 index 0000000..b45c735 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/expected_snr_values.py @@ -0,0 +1,192 @@ +'''This module can be used to find expected SNR values for RNO-G stations from a known stable time period.''' +import logging +import os +import numpy as np +from argparse import ArgumentParser +import logging +import sys +from astropy.time import Time +import pandas as pd +import json + +SCRIPT_DIR_REF = os.path.dirname(os.path.abspath(__file__)) +EXPECTED_VALUES_DIR_REF = os.path.join(SCRIPT_DIR_REF, "expected_snr") +PLOTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "plots_reference", "snr") +LOGS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "logs_reference", "snr") +RESULTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "results_reference", "snr") + +os.makedirs(PLOTS_DIR_REF, exist_ok=True) +os.makedirs(LOGS_DIR_REF, exist_ok=True) +os.makedirs(RESULTS_DIR_REF, exist_ok=True) +os.makedirs(EXPECTED_VALUES_DIR_REF, exist_ok=True) + +PARENT_DIR = os.path.dirname(SCRIPT_DIR_REF) +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") +sys.path.insert(0, PARENT_DIR) + +from analysis_functions_sva.z_score_analysis_sva import calculate_statistics_log_paramater, calculate_z_score_parameter, symmetry_metrics_channel_z_score, symmetry_metrics_z_score, find_k_value, save_values_json, load_values_json, outlier_flag, find_outlier_details +from plotting_functions_sva.plotting_sva_snr import choose_day_interval, plot_snr_against_time +from helper_functions.read_rnog_runtable import read_rnog_runtable +from monitoring_data_functions_sva.get_monitoring_data_uproot import read_multiple_runs, choose_trigger_type_header +from helper_functions.output_writer import write_snr_outlier_details, write_failed_runs_to_csv +from helper_functions.config_helper import get_station_config + +logger = logging.getLogger(__name__) + +def setup_logging(station_id, run_label): + + log_file = os.path.join(LOGS_DIR_REF, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +def metadata_dict(station_id, first_run, last_run, times, trigger_type="FORCE", comment=""): + '''Create a metadata dictionary to save with the expected values, containing station ID, run numbers and time period.''' + if isinstance(times, Time): + start_time = times.min().iso + end_time = times.max().iso + else: + start_time = str(np.min(times)) + end_time = str(np.max(times)) + + metadata = { + "station_id": station_id, + "run_range": f"{first_run} - {last_run}", + "excluded_runs": args.exclude_runs if args.exclude_runs else None, + "n_events": len(times), + "trigger_type": trigger_type, + "start_time": start_time, + "end_time": end_time, + "comment": comment + } + logger.info(f"Metadata for expected values: {metadata}") + return metadata + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis - Expected SNR values.") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + argparser.add_argument("--save-values", action="store_true", help="Whether to save the calculated reference values as JSON files in the script directory, e.g. --save-values") + argparser.add_argument("--data_location", type=str, default="desy", help="Location of the data. Use 'desy' (inbox data), 'uchicago' (mirrored data) or provide a custom path to the data directory, e.g. --data_location /path/to/data") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + args = argparser.parse_args() + + station_id = args.station_id + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + setup_logging(station_id, run_label) + + logger.info("Using monitoring method to read data") + + # Choose the data location based on the argument provided + if args.data_location == "desy": + logger.info("Using DESY inbox data location for the analysis.") + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + elif args.data_location == "uchicago": + logger.info("Using UChicago mirrored data location for the analysis.") + base_data_path = "/data/satellite" + else: + logger.info(f"Using custom data location {args.data_location} for the analysis.") + base_data_path = args.data_location + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + all_channels = config["all_channels"] + + # Choose RADIANT or DIDAQ based on the station configuration (FORCE trigger name is the same for both) + digitizer_type = config["daq_type"] + if digitizer_type not in ["radiant", "didaq"]: + logger.error(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'.") + raise ValueError(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'. Please check the station configuration in config_station.json.") + + combined_event_info = read_multiple_runs(base_path = base_data_path, station_id = station_id, run_numbers=run_numbers, daq_type=digitizer_type) + snr_arr = combined_event_info["snr_arr"] + trigger_type_arr = combined_event_info["triggerType"] + times = combined_event_info["trigger_time_utc"] + run_no = combined_event_info["run_no"] + event_number_arr = combined_event_info["event_number_arr"] + failed_run_info = combined_event_info["failed_run_info"] or {} + failed_runs = list(failed_run_info.keys()) + + force_mask = choose_trigger_type_header(trigger_type_arr, "FORCE", digitizer_type) + run_no_force = run_no[force_mask] + event_number_force = event_number_arr[force_mask] + + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=RESULTS_DIR_REF) + + logger.info("Start calculating expected values for SNR parameter...") + times_force = times[force_mask] + snr_arr_force = snr_arr[:, force_mask] + log_snr_arr, log_mean_list, log_median_list, log_std_list, log_difference_list = calculate_statistics_log_paramater(snr_arr_force) + z_score_arr_log_snr = calculate_z_score_parameter(log_snr_arr, log_mean_list, log_std_list, all_channels) + + k_values_log_snr = find_k_value(z_score_arr_log_snr, all_channels, quantile=0.999) + metadata = metadata_dict(station_id, first_run, last_run, times_force, trigger_type="FORCE", comment="") + if args.save_values: + logger.info("Saving expected values for SNR as JSON files...") + save_values_json(k_values_log_snr, log_mean_list, log_std_list, filename=f"expected_snr_values_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata) + + flag_outliers_snr = outlier_flag(z_score_arr_log_snr, k_values_log_snr, all_channels) + + outlier_details_snr = find_outlier_details(z_score_arr_log_snr, k_values_log_snr, flag_outliers_snr, all_channels, run_no_force, event_number_force) + write_snr_outlier_details(outlier_details_snr, station_id, run_label, n_events_force = len(times_force), results_dir=RESULTS_DIR_REF) + + day_interval = choose_day_interval(times) + plot_snr_against_time(station_id, times_force, snr_arr_force, flag_outliers_snr, z_score_arr_log_snr, k_values_log_snr, all_channels, PLOTS_DIR_REF, run_label, nrows=12, ncols=2, day_interval=day_interval) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/README.md b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/README.md new file mode 100644 index 0000000..96dc047 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/README.md @@ -0,0 +1 @@ +The scripts in this directory can be used to calculate the expected values, using dataProviderRNOG() module to read data. The current version uses monitoring.root data. These scripts won't be updated anymore. \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_rms_values_dataproviderrnog.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_rms_values_dataproviderrnog.py new file mode 100644 index 0000000..93d34ea --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_rms_values_dataproviderrnog.py @@ -0,0 +1,204 @@ +'''This module can be used to find expected RMS (or Vrms) values for RNO-G stations from a known stable time period.''' +import logging +import os +import numpy as np +from argparse import ArgumentParser +import json +import sys +import pandas as pd +from NuRadioReco.utilities import units + +SCRIPT_DIR_REF = os.path.dirname(os.path.abspath(__file__)) +PARENT_DIR = os.path.dirname(os.path.dirname(SCRIPT_DIR_REF)) +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") +sys.path.insert(0, PARENT_DIR) + +logger = logging.getLogger(__name__) + +from analysis_functions_sva.z_score_analysis_sva import outlier_details, calculate_z_score_parameter, find_k_value, save_values_json, outlier_flag, find_outlier_details, calculate_z_score_rolling, metadata_dict, calculate_expected_values_per_trigger +from plotting_functions_sva.plotting_sva_vrms import plot_vrms_values_against_time_single_trigger_zscore, plot_rolling_mean_std, plot_rolling_mean_linregress, create_heatmap_plot +from monitoring_data_functions_sva.get_monitoring_data_uproot import read_multiple_runs, choose_trigger_type_header +from analysis_functions_sva.vrms_analysis_sva import get_rms_per_trigger_monitoring, calculate_vrms +from analysis_functions_sva.vrms_stability_analysis_sva import get_rms_per_run, relative_median_shift, decision_metric +from helper_functions.read_rnog_runtable import read_rnog_runtable +from sva_dataproviderrnog.read_rnog_data_nuradio import read_rnog_data +from sva_dataproviderrnog.science_verification_analysis_dataprovider import choose_trigger_type +from helper_functions.output_writer import write_failed_runs_to_csv, write_vrms_outlier_details +from helper_functions.config_helper import get_station_config + +def setup_logging(station_id, run_label): + + log_file = os.path.join(LOGS_DIR_REF, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis - Expected RMS values. !!!! Outdated !!!!") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("-b", "--backend", type=str, default="pyroot", help="!!! Only needed for method 'dataProviderRNOG' !!!. Backend to use for reading data, should be either pyroot or uproot (default: pyroot), e.g. --backend pyroot or --backend uproot") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + argparser.add_argument("--sampling_rate", type=str, default= "after_2024", choices=["before_2024", "after_2024"], help="!!! Only needed for method 'monitoring' !!!. Sampling rate to use, choices are 'before_2024' (3.2 GHz) and 'after_2024' (2.4 GHz), default is 'after_2024'.") + argparser.add_argument("--save-values", action="store_true", help="Whether to save the calculated reference values as JSON files in the script directory, e.g. --save-values") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + args = argparser.parse_args() + + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + + use_monitoring = False + parameter_label = "vrms" + logger.info("Using dataProviderRNOG method to read data") + + station_id = args.station_id + backend = args.backend + if backend not in ["pyroot", "uproot"]: + raise ValueError("Backend should be either 'pyroot' or 'uproot'") + + sampling_rate_choice = args.sampling_rate + sampling_rate = {"after_2024": 2.4*units.GHz, + "before_2024": 3.2*units.GHz} + sr = sampling_rate[sampling_rate_choice] + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + EXPECTED_VALUES_DIR_REF = os.path.join(SCRIPT_DIR_REF, f"expected_{parameter_label}") + PLOTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, f"plots_reference/{parameter_label}") + LOGS_DIR_REF = os.path.join(SCRIPT_DIR_REF, f"logs_reference/{parameter_label}") + RESULTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, f"results_reference/{parameter_label}") + + os.makedirs(PLOTS_DIR_REF, exist_ok=True) + os.makedirs(LOGS_DIR_REF, exist_ok=True) + os.makedirs(RESULTS_DIR_REF, exist_ok=True) + os.makedirs(EXPECTED_VALUES_DIR_REF, exist_ok=True) + + setup_logging(station_id, run_label) + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + all_channels = config["all_channels"] + + spec_arr, trace_arr, times_trace_arr, snr_arr, run_no, times, freqs, event_info, glitch_arr, block_offsets_arr = read_rnog_data(station_id, run_numbers, backend=backend, sampling_rate=sr) + + force_mask = choose_trigger_type(event_info, "FORCE") + lt_mask = choose_trigger_type(event_info, "LT") + radiant0_mask = choose_trigger_type(event_info, "RADIANT0") + radiant1_mask = choose_trigger_type(event_info, "RADIANT1") + + times = np.array(times) + run_no_force = event_info["run"][force_mask] + event_number_force = event_info["eventNumber"][force_mask] + + run_no_radiant0 = event_info["run"][radiant0_mask] + event_number_radiant0 = event_info["eventNumber"][radiant0_mask] + + run_no_radiant1 = event_info["run"][radiant1_mask] + event_number_radiant1 = event_info["eventNumber"][radiant1_mask] + + run_no_lt = event_info["run"][lt_mask] + event_number_lt = event_info["eventNumber"][lt_mask] + failed_run_info = {} + + vrms_arr, vrms_arr_force, vrms_arr_radiant0, vrms_arr_radiant1, vrms_arr_lt = calculate_vrms(trace_arr, event_info) + + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=RESULTS_DIR_REF) + + logger.info(f"Start calculating expected values for {parameter_label} parameter...") + times_force = times[force_mask] + times_radiant0 = times[radiant0_mask] + times_radiant1 = times[radiant1_mask] + times_lt = times[lt_mask] + + z_score_force, z_score_rolling_force, k_values_force, vrms_mean_force, vrms_std_force, metadata_force, rolling_mean_force, rolling_std_force = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_force, times_force, trigger_type="FORCE", excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_radiant0, z_score_rolling_radiant0, k_values_radiant0, vrms_mean_radiant0, vrms_std_radiant0, metadata_radiant0, rolling_mean_radiant0, rolling_std_radiant0 = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_radiant0, times_radiant0, trigger_type="RADIANT0", excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_radiant1, z_score_rolling_radiant1, k_values_radiant1, vrms_mean_radiant1, vrms_std_radiant1, metadata_radiant1, rolling_mean_radiant1, rolling_std_radiant1 = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_radiant1, times_radiant1, trigger_type="RADIANT1", excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + z_score_lt, z_score_rolling_lt, k_values_lt, vrms_mean_lt, vrms_std_lt, metadata_lt, rolling_mean_lt, rolling_std_lt = calculate_expected_values_per_trigger(station_id, first_run, last_run, vrms_arr_lt, times_lt, trigger_type="LT", excluded_runs=excluded_runs, run_no=run_no, all_channels=all_channels) + + if args.save_values: + save_values_json(k_values_force, vrms_mean_force, vrms_std_force, filename=f"expected_{parameter_label}_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_force) + # save_values_json(k_values_radiant0, vrms_mean_radiant0, vrms_std_radiant0, filename=f"expected_{parameter_label}_radiant0_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_radiant0) + # save_values_json(k_values_radiant1, vrms_mean_radiant1, vrms_std_radiant1, filename=f"expected_{parameter_label}_radiant1_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_radiant1) + # save_values_json(k_values_lt, vrms_mean_lt, vrms_std_lt, filename=f"expected_{parameter_label}_lt_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata_lt) + + flag_outliers_force, outlier_details_force = outlier_details(z_score_force, k_values_force, all_channels, run_no_force, event_number_force, trigger_label="FORCE") + flag_outliers_radiant0, outlier_details_radiant0 = outlier_details(z_score_radiant0, k_values_radiant0, all_channels, run_no_radiant0, event_number_radiant0, trigger_label="RADIANT0") + flag_outliers_radiant1, outlier_details_radiant1 = outlier_details(z_score_radiant1, k_values_radiant1, all_channels, run_no_radiant1, event_number_radiant1, trigger_label="RADIANT1") + flag_outliers_lt, outlier_details_lt = outlier_details(z_score_lt, k_values_lt, all_channels, run_no_lt, event_number_lt, trigger_label="LT") + + k_values_rolling = {int(ch): 4 for ch in all_channels} # Placeholder, as k-values for rolling z-score are not calculated in this script, but could be implemented in the future if needed + flag_outliers_force_rolling, outlier_details_force_rolling = outlier_details(z_score_rolling_force, k_values_rolling, all_channels, run_no_force, event_number_force, trigger_label="FORCE_rolling") + + write_vrms_outlier_details(outlier_details_force, station_id, run_label, trigger_label="FORCE", n_events = len(times_force), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_radiant0, station_id, run_label, trigger_label="RADIANT0", n_events = len(times_radiant0), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_radiant1, station_id, run_label, trigger_label="RADIANT1", n_events = len(times_radiant1), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_lt, station_id, run_label, trigger_label="LT", n_events = len(times_lt), results_dir=RESULTS_DIR_REF) + write_vrms_outlier_details(outlier_details_force_rolling, station_id, run_label, trigger_label="FORCE_rolling", n_events = len(times_force), results_dir=RESULTS_DIR_REF) + + plot_vrms_values_against_time_single_trigger_zscore(times_force, vrms_arr_force, flag_outliers_force, z_score_force, k_values_force, trigger_name="FORCE", channel_list=all_channels, station_id=station_id, run_label=run_label, save_location=PLOTS_DIR_REF, use_monitoring=True) + rms_arr_per_run_dict_force = get_rms_per_run(vrms_arr_force, run_no_force) + + relative_median_shift_results = relative_median_shift(rms_arr_per_run_dict_force, all_channels) + + with open(os.path.join(RESULTS_DIR_REF, f"rms_relative_median_shift_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(relative_median_shift_results, f, indent=4) + + create_heatmap_plot(relative_median_shift_results, label = "Relative Median Shift", save_dir = PLOTS_DIR_REF, channel_list=all_channels, station_id = station_id,matrix_key = "median_shift_matrix", run_label=run_label, cmap="Reds") + + rms_results = decision_metric(outlier_details_force, relative_median_shift_results, n_events_force=len(times_force), channels=all_channels) + with open(os.path.join(RESULTS_DIR_REF, f"rms_stability_decision_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(rms_results, f, indent=4) diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_snr_values_dataproviderrnog.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_snr_values_dataproviderrnog.py new file mode 100644 index 0000000..51ab0b2 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated/expected_snr_values_dataproviderrnog.py @@ -0,0 +1,182 @@ +'''This module can be used to find expected SNR values for RNO-G stations from a known stable time period.''' +import logging +import os +import numpy as np +from argparse import ArgumentParser +import logging +import sys +from astropy.time import Time +import pandas as pd +from NuRadioReco.utilities import units +import json + +SCRIPT_DIR_REF = os.path.dirname(os.path.abspath(__file__)) +EXPECTED_VALUES_DIR_REF = os.path.join(SCRIPT_DIR_REF, "expected_snr") +PLOTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "plots_reference", "snr") +LOGS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "logs_reference", "snr") +RESULTS_DIR_REF = os.path.join(SCRIPT_DIR_REF, "results_reference", "snr") +os.makedirs(PLOTS_DIR_REF, exist_ok=True) +os.makedirs(LOGS_DIR_REF, exist_ok=True) +os.makedirs(RESULTS_DIR_REF, exist_ok=True) +os.makedirs(EXPECTED_VALUES_DIR_REF, exist_ok=True) + +PARENT_DIR = os.path.dirname(os.path.dirname(SCRIPT_DIR_REF)) +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") +sys.path.insert(0, PARENT_DIR) + +from analysis_functions_sva.z_score_analysis_sva import calculate_statistics_log_paramater, calculate_z_score_parameter, symmetry_metrics_channel_z_score, symmetry_metrics_z_score, find_k_value, save_values_json, load_values_json, outlier_flag, find_outlier_details +from plotting_functions_sva.plotting_sva_snr import choose_day_interval, plot_snr_against_time +from helper_functions.read_rnog_runtable import read_rnog_runtable +from sva_dataproviderrnog.read_rnog_data_nuradio import read_rnog_data +from sva_dataproviderrnog.science_verification_analysis_dataprovider import choose_trigger_type +from monitoring_data_functions_sva.get_monitoring_data_uproot import read_multiple_runs, choose_trigger_type_header +from helper_functions.output_writer import write_snr_outlier_details, write_failed_runs_to_csv +from helper_functions.config_helper import get_station_config + +logger = logging.getLogger(__name__) + +def setup_logging(station_id, run_label): + + log_file = os.path.join(LOGS_DIR_REF, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +def metadata_dict(station_id, first_run, last_run, times, trigger_type="FORCE", comment=""): + '''Create a metadata dictionary to save with the expected values, containing station ID, run numbers and time period.''' + if isinstance(times, Time): + start_time = times.min().iso + end_time = times.max().iso + else: + start_time = str(np.min(times)) + end_time = str(np.max(times)) + + metadata = { + "station_id": station_id, + "run_range": f"{first_run} - {last_run}", + "excluded_runs": args.exclude_runs if args.exclude_runs else None, + "n_events": len(times), + "trigger_type": trigger_type, + "start_time": start_time, + "end_time": end_time, + "comment": comment + } + logger.info(f"Metadata for expected values: {metadata}") + return metadata + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis - Expected SNR values. !!!! Outdated !!!!") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("-b", "--backend", type=str, default="pyroot", help="!!! Only needed for method 'monitoring' !!!. Backend to use for reading data, should be either pyroot or uproot (default: pyroot), e.g. --backend pyroot or --backend uproot") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + argparser.add_argument("--sampling_rate", type=str, default= "after_2024", choices=["before_2024", "after_2024"], help="!!! Only needed for method 'monitoring' !!!. Sampling rate to use, choices are 'before_2024' (3.2 GHz) and 'after_2024' (2.4 GHz), default is 'after_2024'.") + argparser.add_argument("--save-values", action="store_true", help="Whether to save the calculated reference values as JSON files in the script directory, e.g. --save-values") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + args = argparser.parse_args() + + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + + logger.info("Using dataProviderRNOG method to read data") + + station_id = args.station_id + backend = args.backend + if backend not in ["pyroot", "uproot"]: + raise ValueError("Backend should be either 'pyroot' or 'uproot'") + + sampling_rate_choice = args.sampling_rate + sampling_rate = {"after_2024": 2.4*units.GHz, + "before_2024": 3.2*units.GHz} + sr = sampling_rate[sampling_rate_choice] + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + setup_logging(station_id, run_label) + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + all_channels = config["all_channels"] + + spec_arr, trace_arr, times_trace_arr, snr_arr, run_no, times, freqs, event_info, glitch_arr, block_offsets_arr = read_rnog_data(station_id, run_numbers, backend=backend, sampling_rate=sr) + force_mask = choose_trigger_type(event_info, "FORCE") + times = np.array(times) + run_no_force = event_info["run"][force_mask] + event_number_force = event_info["eventNumber"][force_mask] + failed_run_info = {} + + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=RESULTS_DIR_REF) + + logger.info("Start calculating expected values for SNR parameter...") + times_force = times[force_mask] + snr_arr_force = snr_arr[:, force_mask] + log_snr_arr, log_mean_list, log_median_list, log_std_list, log_difference_list = calculate_statistics_log_paramater(snr_arr_force) + z_score_arr_log_snr = calculate_z_score_parameter(log_snr_arr, log_mean_list, log_std_list, all_channels) + + k_values_log_snr = find_k_value(z_score_arr_log_snr, all_channels, quantile=0.999) + metadata = metadata_dict(station_id, first_run, last_run, times_force, trigger_type="FORCE", comment="") + if args.save_values: + logger.info("Saving expected values for SNR as JSON files...") + save_values_json(k_values_log_snr, log_mean_list, log_std_list, filename=f"expected_snr_values_station{station_id}.json", SCRIPT_DIR=EXPECTED_VALUES_DIR_REF, metadata=metadata) + + flag_outliers_snr = outlier_flag(z_score_arr_log_snr, k_values_log_snr, all_channels) + + outlier_details_snr = find_outlier_details(z_score_arr_log_snr, k_values_log_snr, flag_outliers_snr, all_channels, run_no_force, event_number_force) + write_snr_outlier_details(outlier_details_snr, station_id, run_label, n_events_force = len(times_force), results_dir=RESULTS_DIR_REF) + + day_interval = choose_day_interval(times) + plot_snr_against_time(station_id, times_force, snr_arr_force, flag_outliers_snr, z_score_arr_log_snr, k_values_log_snr, all_channels, PLOTS_DIR_REF, run_label, nrows=12, ncols=2, day_interval=day_interval) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/config_helper.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/config_helper.py new file mode 100644 index 0000000..56d857e --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/config_helper.py @@ -0,0 +1,6 @@ +def get_station_config(station_id, default_station_config, station_specific_adjustments): + cfg = default_station_config.copy() + if str(station_id) in station_specific_adjustments: + adjustments = station_specific_adjustments[str(station_id)] + cfg.update(adjustments) + return cfg diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/output_writer.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/output_writer.py new file mode 100644 index 0000000..9c37c76 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/output_writer.py @@ -0,0 +1,642 @@ +import os +import csv +import logging +import textwrap +import pandas as pd + +logger = logging.getLogger(__name__) + + +def write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir): + failed_runs_file = os.path.join(results_dir, f"station{station_id}_failed_runs_in_runrange_{run_label}.csv") + with open(failed_runs_file, "w", newline="") as f: + writer = csv.writer(f) + writer.writerow(["Run Number", "Reason for Failure"]) + for run_no, reason in failed_run_info.items(): + writer.writerow([run_no, reason]) + logger.warning(f"Failed to process some runs for station {station_id}: {list(failed_run_info.keys())}. Information about these runs has been written to {failed_runs_file} in the {results_dir} directory. Please check the file for details as this might indicate potential issues!") + +def write_spectral_results(all_excess_info, channels, station_id, run_label, results_dir): + # dCache/pnfs enforces write-once semantics: a file can only be opened for + # writing once, so all channels must be written in a single open() call. + spectral_results_file = os.path.join(results_dir, f"spectral_analysis_results_{station_id}_{run_label}.txt") + + content = "" + for ch in channels: + content += f"Channel {ch:02d}:\n" + for band, results in all_excess_info[ch].items(): + content += f"\n=== {band} ===\n" + for key, value in results.items(): + content += f"{key}: {value}\n" + content += "\n" + + with open(spectral_results_file, "w") as f: + f.write(content) + logger.info(f"Spectral analysis results written to {spectral_results_file}") + +def write_snr_outlier_details(outlier_details, station_id, run_label, n_events_force, results_dir): + outlier_results_file = os.path.join(results_dir, f"force_snr_details_{station_id}_{run_label}.txt") + with open(outlier_results_file, "w") as f: + for ch in sorted(outlier_details.keys()): + entries = outlier_details[ch] + n_outliers = len(entries) + + if n_outliers == 0: + f.write(f"\nChannel {ch:02d}:\n 0 outliers\n") + continue + + k_ch = entries[0]["k"] + outlier_fraction = n_outliers / n_events_force if n_events_force > 0 else 0.0 + max_delta = max(abs(e.get("z_minus_k", 0.0)) for e in entries) + f.write(f"\nChannel {ch:02d}:\n {n_outliers} outliers (k = {k_ch:.2f}), outlier fraction: {outlier_fraction:.2f}, max delta: {max_delta:.2f}\n") + + for e in entries: + f.write(f" - run {e['run']}, event {e['eventNumber']}, |z| = {e['z_abs']:.2f} (delta = {e['z_minus_k']:.2f} above k)\n") + + logger.info(f"SNR outlier details written to {outlier_results_file}") + +def write_vrms_outlier_details(outlier_details, station_id, run_label, trigger_label, n_events, results_dir, use_monitoring = False): + if use_monitoring: + file_label = "rms" + else: + file_label = "vrms" + + outlier_results_file = os.path.join(results_dir, f"{file_label}_details_{station_id}_{run_label}_{trigger_label}.txt") + with open(outlier_results_file, "w") as f: + for ch in sorted(outlier_details.keys()): + entries = outlier_details[ch] + n_outliers = len(entries) + + if n_outliers == 0: + f.write(f"\nChannel {ch:02d}:\n 0 outliers\n") + continue + + k_ch = entries[0]["k"] + outlier_fraction = n_outliers / n_events if n_events > 0 else 0.0 + max_delta = max(abs(e.get("z_minus_k", 0.0)) for e in entries) + f.write(f"\nChannel {ch:02d}:\n {n_outliers} outliers (k = {k_ch:.2f}), outlier fraction: {outlier_fraction:.2f}, max delta: {max_delta:.2f}\n") + + for e in entries: + f.write(f" - run {e['run']}, event {e['eventNumber']}, |z| = {e['z_abs']:.2f} (delta = {e['z_minus_k']:.2f} above k)\n") + + logger.info(f"{file_label.capitalize()} outlier details written to {outlier_results_file}") + +def write_vrms_modality_results(modality, tail_label, trigger_label, station_id, run_label, results_dir, use_monitoring = False): + if use_monitoring: + file_label = "rms" + else: + file_label = "vrms" + + modality_results_file = os.path.join(results_dir, f"{file_label}_modality_{station_id}_{run_label}_{trigger_label}.txt") + with open(modality_results_file, "w") as f: + for ch in sorted(modality.keys()): + modality_result = modality[ch] + tail_label_result = tail_label[ch] + f.write(f"Channel {ch} ({trigger_label} events): {modality_result} ({tail_label_result})\n") + logger.info(f"{file_label.capitalize()} modality results for {trigger_label} events written to {modality_results_file}") + +def write_glitching_results(glitch_info, station_id, run_label, all_channels, results_dir): + glitch_results_file = os.path.join(results_dir, f"glitching_analysis_results_{station_id}_{run_label}.txt") + lines = [ + ( + f"Channel {ch:2d} | " + f"n_glitches: {glitch_info[ch]['n_glitches']:4d} | " + f"n_events: {glitch_info[ch]['n_events']:<4d} | " + f"(frac={glitch_info[ch]['glitch_fraction']:.3f}) | " + f"p={glitch_info[ch]['pval']:.2e} | " + f"CI99%={glitch_info[ch]['confidence_interval']} | " + f"{glitch_info[ch]['validation']}" + ) + for ch in all_channels] + with open(glitch_results_file, "w") as f: + f.write("\n".join(lines)) + logger.info(f"Glitching analysis results written to {glitch_results_file}") + +def block_offset_channel_health(block_offset_stats, ref_block_off_dict, use_monitoring=False): + + results_dict = {} + for ch in sorted(block_offset_stats.keys()): + stats = block_offset_stats[ch] + results = ref_block_off_dict.get(str(ch), {}) + if results == {}: + results_dict[ch] = "?" + continue + + if use_monitoring: + median_ref = results.get("median_adc_offset_counts", None) + p99_ref = results.get("p99_adc_offset_counts", None) + if median_ref is not None and stats["median"] > median_ref: + results_dict[ch] = "X" + elif p99_ref is not None and stats["p99"] > p99_ref: + results_dict[ch] = "X" + else: + results_dict[ch] = "OK" + else: + median_ref = results.get("median_adc_offset_mv", None) + p99_ref = results.get("p99_adc_offset_mv", None) + if median_ref is not None and stats["before_median"] > median_ref: + results_dict[ch] = "X" + elif p99_ref is not None and stats["p99_before"] > p99_ref: + results_dict[ch] = "X" + elif median_ref is not None and stats["after_median"] > median_ref: + results_dict[ch] = "X" + elif p99_ref is not None and stats["p99_after"] > p99_ref: + results_dict[ch] = "X" + else: + results_dict[ch] = "OK" + + return results_dict + +def write_block_offset_results(block_offset_stats, station_id, run_label, ref_block_off_dict, results_dir, use_monitoring=False): + block_offset_results_file = os.path.join(results_dir, f"block_offset_analysis_results_{station_id}_{run_label}.txt") + # Single source of truth for the OK/X verdict - avoid re-deriving it here so it can't drift out of sync. + results_dict = block_offset_channel_health(block_offset_stats, ref_block_off_dict, use_monitoring=use_monitoring) + with open(block_offset_results_file, "w") as f: + for ch in sorted(block_offset_stats.keys()): + stats = block_offset_stats[ch] + results = ref_block_off_dict.get(str(ch), {}) + if results == {}: + f.write(f"Channel {ch:02d}:\n") + f.write(" No reference block offset data available for this channel.\n") + continue + + if use_monitoring: + median_ref = results.get("median_adc_offset_counts", None) + p99_ref = results.get("p99_adc_offset_counts", None) + f.write(f"Channel {ch:02d}:\n") + f.write(f" Mean block offset: {stats['mean']}, median: {stats['median']}, std: {stats['std']}, IQR: {stats['iqr']}, P99: {stats['p99']}\n") + if results_dict[ch] == "X": + if median_ref is not None and stats["median"] > median_ref: + logger.warning(f"Channel {ch:02d} has a high median block offset of {stats['median']}, which may indicate a potential issue with the channel.") + elif p99_ref is not None and stats["p99"] > p99_ref: + logger.warning(f"Channel {ch:02d} has a high P99 of block offsets ({stats['p99']}), indicating significant variability that may need further investigation.") + + else: + median_ref = results.get("median_adc_offset_mv", None) + p99_ref = results.get("p99_adc_offset_mv", None) + f.write(f"Channel {ch:02d}:\n") + f.write(f" Before removal - mean: {stats['before_mean']} V, median: {stats['before_median']} V, std: {stats['before_std']} V, IQR: {stats['iqr_before']} V, P99: {stats['p99_before']} V\n") + f.write(f" After removal - mean: {stats['after_mean']} V, median: {stats['after_median']} V, std: {stats['after_std']} V, IQR: {stats['iqr_after']} V, P99: {stats['p99_after']} V\n") + f.write(f" Removal fraction (based on median): {stats['removal_fraction']*100:.1f}%\n") + f.write(f" P99 reduction fraction: {stats['p99_reduction_fraction']*100:.1f}%\n") + + if results_dict[ch] == "X": + if median_ref is not None and stats["before_median"] > median_ref: + logger.warning(f"Channel {ch:02d} has a high median block offset of {stats['before_median']} V before removal, which may indicate a potential issue with the channel.") + elif p99_ref is not None and stats["p99_before"] > p99_ref: + logger.warning(f"Channel {ch:02d} has a high P99 of block offsets ({stats['p99_before']} V) before removal, indicating significant variability that may need further investigation.") + elif median_ref is not None and stats["after_median"] > median_ref: + logger.warning(f"Channel {ch:02d} has a relatively high median block offset of {stats['after_median']} V after removal, removal was not fully effective.") + elif p99_ref is not None and stats["p99_after"] > p99_ref: + logger.warning(f"Channel {ch:02d} has a relatively high P99 of block offsets ({stats['p99_after']} V) after removal, indicating that there may still be significant variability in block offsets.") + + logger.info(f"Block offset analysis results written to {block_offset_results_file}") + return results_dict + +def channel_health(row): + severity = {"OK": 0, "!!": 1, "X": 2} + vals = [severity[v] for v in row if v in severity] + if not vals: + return "-" + inv = {0: "OK", 1: "!!", 2: "X"} + return inv[max(vals)] + +def create_result_csv_file(station_id, run_label, n_events_force, surface_channels, downward_channels, upward_channels, all_channels, validation_results, glitch_info, block_offsets_result_dict, rms_results, modality_dict_force, modality_dict_lt, + modality_dict_radiant0, modality_dict_radiant1, outlier_details, csv_dir, rms_label): + out_csv_file = os.path.join(csv_dir, f"validation_summary_station{station_id}_{run_label}.csv") + ch_list = list(all_channels) + + spectral_col = [] + glitch_col = [] + block_offset_col = [] + rms_stability_col = [] + modality_force_col = [] + modality_lt_col = [] + modality_radiant0_col = [] + modality_radiant1_col = [] + snr_col = [] + + for ch in ch_list: + df_spec_val = "" + if ch in surface_channels: + spectral_validation = None + vr = validation_results.get(ch, {}) + spectral_validation = vr.get("galactic_excess", {}) + + if spectral_validation is None: + df_spec_val = "?" + else: + if ch in downward_channels: + if spectral_validation == "NO EXCESS": + df_spec_val = "OK" + elif spectral_validation == "WEAK EXCESS": + df_spec_val = "!!" + elif spectral_validation in ["MODERATE EXCESS", "STRONG EXCESS"]: + df_spec_val = "X" + else: + df_spec_val = "?" + elif ch in upward_channels: + if spectral_validation in ["STRONG EXCESS", "MODERATE EXCESS"]: + df_spec_val = "OK" + elif spectral_validation == "WEAK EXCESS": + df_spec_val = "!!" + elif spectral_validation == "NO EXCESS": + df_spec_val = "X" + else: + df_spec_val = "?" + else: + df_spec_val = "?" + else: + df_spec_val = "-" + + spectral_col.append(df_spec_val) + + # Glitching column + if glitch_info is None: + glitch_val = "-" + glitch_col.append(glitch_val) + else: + info = glitch_info.get(ch, None) + glitch_val_raw = info.get("validation") if info is not None else "-" + if glitch_val_raw == "NO EXCESSIVE GLITCHING": + glitch_val = "OK" + elif glitch_val_raw == "WEAK EXCESSIVE GLITCHING": + glitch_val = "!!" + elif glitch_val_raw in ["MODERATE EXCESSIVE GLITCHING", "STRONG EXCESSIVE GLITCHING"]: + glitch_val = "X" + else: + glitch_val = "-" + glitch_col.append(glitch_val) + + # Vrms analysis column + if rms_results[ch] is None: + rms_val = "-" + rms_stability_col.append(rms_val) + else: + rms_value = rms_results[ch].get("decision", "-") + rms_stability_col.append(rms_value) + + if modality_dict_force is None: + modality_value = "-" + modality_force_col.append(modality_value) + else: + n_peaks = modality_dict_force[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = f"X" + modality_force_col.append(modality_value) + + if modality_dict_lt is None: + modality_value = "-" + modality_lt_col.append(modality_value) + else: + n_peaks = modality_dict_lt[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = f"X" + modality_lt_col.append(modality_value) + + if modality_dict_radiant0 is None: + modality_value = "-" + modality_radiant0_col.append(modality_value) + else: + n_peaks = modality_dict_radiant0[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = "X" + modality_radiant0_col.append(modality_value) + if modality_dict_radiant1 is None: + modality_value = "-" + modality_radiant1_col.append(modality_value) + else: + n_peaks = modality_dict_radiant1[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = "X" + modality_radiant1_col.append(modality_value) + + # SNR validation column + outlier_ch_info = outlier_details.get(ch, []) + n_out = len(outlier_ch_info) + if n_out == 0: + snr_value = "OK" + else: + max_delta = max(abs(o.get("z_minus_k", 0.0)) for o in outlier_ch_info) + frac_out = n_out / n_events_force if n_events_force > 0 else 0.0 + if max_delta < 3.0: + snr_value = "OK" + + elif max_delta < 5.0: + snr_value = "OK" if frac_out < 0.002 else "!!" + + else: # max_delta >= 5 + if n_out == 1 and frac_out < 0.002: + snr_value = "OK" + elif frac_out < 0.004: + snr_value = "!!" + else: + snr_value = "X" + snr_col.append(snr_value) + + # Block offsets column + block_offset_result = block_offsets_result_dict.get(ch, None) + if block_offset_result is None: + block_offset_val = "-" + block_offset_col.append(block_offset_val) + else: + block_offset_val = block_offset_result + block_offset_col.append(block_offset_val) + + df = pd.DataFrame({ + "Channel": ch_list, + "SNR": snr_col, + "Galaxy (FORCE)": spectral_col, + f"{rms_label.capitalize()} Stability (FORCE)": rms_stability_col, + f"{rms_label.capitalize()} (FORCE)": modality_force_col, + f"{rms_label.capitalize()} (LT)": modality_lt_col, + f"{rms_label.capitalize()} (RADIANT0)": modality_radiant0_col, + f"{rms_label.capitalize()} (RADIANT1)": modality_radiant1_col, + "Glitching": glitch_col, + "Block Offsets": block_offset_col + }) + + health_cols =["SNR", "Galaxy (FORCE)", f"{rms_label.capitalize()} Stability (FORCE)", f"{rms_label.capitalize()} (FORCE)", "Glitching"] + df["Channel Health (FORCE)"] = df[health_cols].apply(channel_health, axis=1) + df.to_csv(out_csv_file, index=False) + logger.info(f"Validation summary saved to {out_csv_file}") + + return df + +def create_result_csv_file_didaq(station_id, run_label, n_events_force, surface_channels, downward_channels, upward_channels, all_channels, validation_results, rms_results, modality_dict_force, modality_dict_lt, + modality_dict_radiant0, modality_dict_radiant1, outlier_details, csv_dir, rms_label): + out_csv_file = os.path.join(csv_dir, f"validation_summary_station{station_id}_{run_label}.csv") + ch_list = list(all_channels) + + spectral_col = [] + rms_stability_col = [] + modality_force_col = [] + modality_lt_col = [] + modality_radiant0_col = [] + modality_radiant1_col = [] + snr_col = [] + + for ch in ch_list: + df_spec_val = "" + if ch in surface_channels: + spectral_validation = None + vr = validation_results.get(ch, {}) + spectral_validation = vr.get("galactic_excess", {}) + + if spectral_validation is None: + df_spec_val = "?" + else: + if ch in downward_channels: + if spectral_validation == "NO EXCESS": + df_spec_val = "OK" + elif spectral_validation == "WEAK EXCESS": + df_spec_val = "!!" + elif spectral_validation in ["MODERATE EXCESS", "STRONG EXCESS"]: + df_spec_val = "X" + else: + df_spec_val = "?" + elif ch in upward_channels: + if spectral_validation in ["STRONG EXCESS", "MODERATE EXCESS"]: + df_spec_val = "OK" + elif spectral_validation == "WEAK EXCESS": + df_spec_val = "!!" + elif spectral_validation == "NO EXCESS": + df_spec_val = "X" + else: + df_spec_val = "?" + else: + df_spec_val = "?" + else: + df_spec_val = "-" + + spectral_col.append(df_spec_val) + + # Vrms analysis column + if rms_results[ch] is None: + rms_val = "-" + rms_stability_col.append(rms_val) + else: + rms_value = rms_results[ch].get("decision", "-") + rms_stability_col.append(rms_value) + + if modality_dict_force is None: + modality_value = "-" + modality_force_col.append(modality_value) + else: + n_peaks = modality_dict_force[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = f"X" + modality_force_col.append(modality_value) + + if modality_dict_lt is None: + modality_value = "-" + modality_lt_col.append(modality_value) + else: + n_peaks = modality_dict_lt[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = f"X" + modality_lt_col.append(modality_value) + + if modality_dict_radiant0 is None: + modality_value = "-" + modality_radiant0_col.append(modality_value) + else: + n_peaks = modality_dict_radiant0[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = "X" + modality_radiant0_col.append(modality_value) + if modality_dict_radiant1 is None: + modality_value = "-" + modality_radiant1_col.append(modality_value) + else: + n_peaks = modality_dict_radiant1[ch]["n_peaks"] + if n_peaks == 0: + modality_value = "!!" + elif n_peaks == 1: + modality_value = "OK" + elif n_peaks == 2: + modality_value = "X" + else: + modality_value = "X" + modality_radiant1_col.append(modality_value) + + # SNR validation column + outlier_ch_info = outlier_details.get(ch, []) + n_out = len(outlier_ch_info) + if n_out == 0: + snr_value = "OK" + else: + max_delta = max(abs(o.get("z_minus_k", 0.0)) for o in outlier_ch_info) + frac_out = n_out / n_events_force if n_events_force > 0 else 0.0 + if max_delta < 3.0: + snr_value = "OK" + + elif max_delta < 5.0: + snr_value = "OK" if frac_out < 0.002 else "!!" + + else: # max_delta >= 5 + if n_out == 1 and frac_out < 0.002: + snr_value = "OK" + elif frac_out < 0.004: + snr_value = "!!" + else: + snr_value = "X" + snr_col.append(snr_value) + + + df = pd.DataFrame({ + "Channel": ch_list, + "SNR": snr_col, + "Galaxy (FORCE)": spectral_col, + f"{rms_label.capitalize()} Stability (FORCE)": rms_stability_col, + f"{rms_label.capitalize()} (FORCE)": modality_force_col, + f"{rms_label.capitalize()} (DEEP PHASED)": modality_lt_col, + f"{rms_label.capitalize()} (SURF UP)": modality_radiant0_col, + f"{rms_label.capitalize()} (SURF DOWN)": modality_radiant1_col, + }) + + health_cols =["SNR", "Galaxy (FORCE)", f"{rms_label.capitalize()} Stability (FORCE)", f"{rms_label.capitalize()} (FORCE)"] + df["Channel Health (FORCE)"] = df[health_cols].apply(channel_health, axis=1) + df.to_csv(out_csv_file, index=False) + logger.info(f"Validation summary saved to {out_csv_file}") + + return df + +def write_readme_for_shifters(readme_file, station_id, run_numbers, times, run_label): + + text = f"""\ + ================================================================================ + README FOR SHIFTERS - Station {station_id}, Runs: {run_numbers} + Time Range: {times[0]} to {times[-1]} + ================================================================================ + + IMPORTANT: This README is intended for shifters to understand the results of + the channel health analysis. Please read it carefully before interpreting the + results. + + This directory contains the results of the channel health analysis for + station {station_id} over the specified runs. The analysis includes spectral + analysis, SNR analysis, Vrms analysis, glitching and block offset analysis + (for RADIANT digitizer type), and generates various plots and summary files, + which will be explained below. + + WHAT TO DO AS A SHIFTER + -------------------------------------------------------------------------------- + 1. Check the Summary CSV File + + The summary CSV file (channel_health_summary.csv) contains the overall + health status of each channel based on the various tests performed. The + columns indicate the results of each test: + X - Test failed with serious deviation from the expected behavior. + !! - Test failed with minor deviation from the expected behavior. + OK - Channel passed the test. + The overall channel health is determined based on the results of the + FORCE trigger tests, and is given in the column "Channel Health (FORCE)". + This combines the results of the SNR, Galaxy, Vrms stability, and Vrms + modality tests for the FORCE trigger (glitching as well for stations with + RADIANT digitizer type - 11, 12, 13, 14, 21, 22, 23, 24). + If a channel fails any of these tests, it is marked as "X" in the overall + health column. If it passes all tests, it is marked as "OK". If it has + minor issues, it may be marked as "!!". + YOU CAN CHECK THE "Channel Health (FORCE)" COLUMN FIRST TO QUICKLY IDENTIFY + CHANNELS THAT NEED ATTENTION AND INVESTIGATE INDIVIDUAL TEST COLUMNS IF + NEEDED. + + 2. Look at the Standard Plots + + The standard plots (plots/standard_plots/) are always produced, regardless + of whether any channel failed a test. They give a quick visual overview of + the station's behavior for the time period: normalized/unnormalized + surface and deep spectra, SNR vs. time, Vrms vs. time, and trigger rates + over time. It's a good idea to skim these even if the summary CSV shows + all channels as "OK". + + 3. Investigate Flagged Channels + + If the summary CSV flags a channel as "X" or "!!", check + plots/failed_test_plots/ and test_results/failed_test_results/ first: + these directories only get populated for tests that failed for at least + one channel (e.g. SNR/z-score distributions, galaxy ratio distributions, + Vrms distributions with the modality fit, glitching, block offsets), so + you can go straight to the relevant plot/report for the failing test. + If a test passed for all channels, its plots and detailed numbers are + still saved (nothing is discarded), just under plots/detailed_plots/ and + test_results/detailed_results/ instead. + + 4. Always Check the Failed-Runs File + + If any of the requested runs could not be read (missing/corrupt files, + invalid timestamps, or runs you excluded with -ex/--exclude-runs), they + are listed with a reason in + test_results/station{station_id}_failed_runs_in_runrange_{run_label}.csv. + This file is only created when at least one run failed, but you should + always check for its presence, as it means the analysis is based on fewer + runs than requested and might indicate issues with the data. + + 5. Check the Log File + + The log file + (logs/logging_science_verification_analysis_station{station_id}_{run_label}.log) + records warnings about missing/invalid runs, low event counts, and + borderline channels that may not be fully captured by the summary CSV. + + 6. Report on the Monday Call + + If you are a shifter, please report any channels that are flagged as "X" + or "!!" in the summary CSV, and any runs that failed to be read, on the + Monday shifter call. This helps the station experts and data quality team + to investigate and address any issues. + + If you are unsure how to interpret a result, or a channel's health looks + suspicious, please reach out to Zeynep Su Selcuk (zeynep.su.selcuk@desy.de). + """ + + with open(readme_file, "w") as f: + f.write(textwrap.dedent(text)) + logger.info(f"!!! README for shifters written to {readme_file}, please check the file for details !!!") + + + + diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/read_rnog_runtable.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/read_rnog_runtable.py new file mode 100644 index 0000000..4b6bfec --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/helper_functions/read_rnog_runtable.py @@ -0,0 +1,8 @@ +import rnog_data.runtable as rt + +#### Runtable query to get run numbers for a given station and time range +def read_rnog_runtable(station_id: int, start_time: str, stop_time: str): + '''Get run numbers from the runtable tool for a given station and time range.''' + RunTable = rt.RunTable() + testrt = RunTable.get_table( start_time=start_time, stop_time=stop_time, stations=[station_id], run_types = ['physics']) + return testrt \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/monitoring_data_functions_sva/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/monitoring_data_functions_sva/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/monitoring_data_functions_sva/get_monitoring_data_uproot.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/monitoring_data_functions_sva/get_monitoring_data_uproot.py new file mode 100644 index 0000000..60e2599 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/monitoring_data_functions_sva/get_monitoring_data_uproot.py @@ -0,0 +1,594 @@ +import uproot +import numpy as np +import os +from tqdm import tqdm +import logging +import json + +#### Script directory for json files +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +PARENT_DIR = os.path.dirname(SCRIPT_DIR) +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") + +# DIDAQ bits +didaq_bits = json.load(open(os.path.join(CONFIG_DIR, "config_station.json"), "r"))["didaq_bits"] + +_DIDAQ_DEEP_PHASED = didaq_bits["DIDAQ_DEEP_PHASED"] +_DIDAQ_SURF_UP = didaq_bits["DIDAQ_SURF_UP"] +_DIDAQ_SURF_DOWN = didaq_bits["DIDAQ_SURF_DOWN"] +_DIDAQ_COINC0 = didaq_bits["DIDAQ_COINC0"] +_DIDAQ_COINC1 = didaq_bits["DIDAQ_COINC1"] + +logger = logging.getLogger(__name__) + +def stack_if_object(branch_data): + arr = np.array(branch_data) + if arr.dtype == object: + if len(arr) == 0: + return np.array([]) # Return an empty array if the input is empty + arr = np.stack(arr) + return arr + +def open_file(path): + if not os.path.isfile(path): + logger.warning(f"File {path} does not exist. Skipping...") # For multiple runs, we can have some missing monitoring files, so we just log a warning and skip those runs instead of raising an error. + return None + return uproot.open(path) + +def get_event_info_from_monitoring_file(file, daq_type): + try: + event_tree = file["events"] + EventSummary = event_tree["EventSummary"] + event_number_arr = stack_if_object(EventSummary["event_number"]) + rms_arr = stack_if_object(EventSummary["rms"]) + max_abs_amplitude_arr = stack_if_object(EventSummary["max_abs_amplitude"]) + + if daq_type == "radiant": + glitching_ts_arr = stack_if_object(EventSummary["glitching_test_statitic"]) # There is typo in the monitoring.root for glitch ts + block_offsets_arr = stack_if_object(EventSummary["block_offset"]) + + return { + "event_number_arr": event_number_arr, # (n_events,) + "rms_arr": rms_arr.T, # (n_ch, n_events) + "max_abs_amplitude_arr": max_abs_amplitude_arr.T, # (n_ch, n_events) + "glitching_test_statistic_arr": glitching_ts_arr.T, # (n_ch, n_events) + "block_offsets_arr": block_offsets_arr.T # (n_ch, n_events) + } + elif daq_type == "didaq": # no glitching test statistic or block offsets for didaq + return { + "event_number_arr": event_number_arr, # (n_events,) + "rms_arr": rms_arr.T, # (n_ch, n_events) + "max_abs_amplitude_arr": max_abs_amplitude_arr.T, # (n_ch, n_events) + } + + except KeyError as e: + logger.error(f"Key {e} not found in events tree of monitoring file. Please check the structure of the monitoring file.") + return None + +def get_run_summary_from_monitoring_file(file): + try: + run_summary = file["RunSummary"] + run_summary_members = run_summary.members + return run_summary_members # is already a dict with member names as keys and arrays as values + + except KeyError as e: + logger.error(f"Key {e} not found in run summary of monitoring file. Please check the structure of the monitoring file.") + return None + +def get_info_from_header_file(header_file, daq_type): + try: + if daq_type == "radiant": + header = header_file["header"] + trigger_time = stack_if_object(header["trigger_time"]) + trigger_time_utc = trigger_time.astype("datetime64[s]") + readout_time = stack_if_object(header["readout_time"]) + duration = np.max(readout_time) - np.min(readout_time) # in seconds + run_no = stack_if_object(header["run_number"]) + event_number = stack_if_object(header["event_number"]) + station_id = stack_if_object(header["station_number"]) + + trigger_info = header["trigger_info"] + force_trigger = stack_if_object(trigger_info["trigger_info.force_trigger"]) + radiant_trigger = stack_if_object(trigger_info["trigger_info.radiant_trigger"]) + lt_trigger = stack_if_object(trigger_info["trigger_info.lt_trigger"]) + which_radiant = stack_if_object(trigger_info["trigger_info.which_radiant_trigger"]) + + # If there are overlaps return None + overlap_mask = (force_trigger.astype(bool) & radiant_trigger.astype(bool)) | (force_trigger.astype(bool) & lt_trigger.astype(bool)) | (radiant_trigger.astype(bool) & lt_trigger.astype(bool)) + if np.any(overlap_mask): + n_overlap = np.sum(overlap_mask) + overlap_event_indices = np.where(overlap_mask)[0] + force_radiant_overlap = np.where(force_trigger.astype(bool) & radiant_trigger.astype(bool))[0] + force_lt_overlap = np.where(force_trigger.astype(bool) & lt_trigger.astype(bool))[0] + radiant_lt_overlap = np.where(radiant_trigger.astype(bool) & lt_trigger.astype(bool))[0] + logger.error(f"Found {n_overlap} events with overlapping trigger types at indices: {overlap_event_indices} for run {run_no[0]}. FORCE-RADIANT: {force_radiant_overlap}, FORCE-LT: {force_lt_overlap}, RADIANT-LT: {radiant_lt_overlap}. Please check the trigger info in the header file.") + return None + + return { + "trigger_time_utc": trigger_time_utc, + "run_no": run_no, + "event_number": event_number, + "station_id": station_id, + "force_trigger": force_trigger, + "radiant_trigger": radiant_trigger, + "lt_trigger": lt_trigger, + "which_radiant": which_radiant, + "readout_time": readout_time, + "duration": duration + } + + elif daq_type == "didaq": + header = header_file["header"] + trigger_time = stack_if_object(header["trigger_time"]) + trigger_time_utc = trigger_time.astype("datetime64[s]") + readout_time = stack_if_object(header["readout_time"]) + duration = np.max(readout_time) - np.min(readout_time) # in seconds + run_no = stack_if_object(header["run_number"]) + event_number = stack_if_object(header["event_number"]) + station_id = stack_if_object(header["station_number"]) + + trigger_info = header["trigger_info"] + force_trigger = stack_if_object(trigger_info["trigger_info.force_trigger"]) + didaq_trigger = stack_if_object(trigger_info["trigger_info.didaq_trigger"]) + didaq_trigger_info_type = stack_if_object(trigger_info["trigger_info.didaq_info.type"]) + + # If there are overlaps return None + overlap_mask = (force_trigger.astype(bool) & didaq_trigger.astype(bool)) + if np.any(overlap_mask): + n_overlap = np.sum(overlap_mask) + overlap_event_indices = np.where(overlap_mask)[0] + logger.error(f"Found {n_overlap} events with overlapping trigger types at indices: {overlap_event_indices} for run {run_no[0]}. Please check the trigger info in the header file.") + return None + + return { + "trigger_time_utc": trigger_time_utc, + "run_no": run_no, + "event_number": event_number, + "station_id": station_id, + "force_trigger": force_trigger, + "didaq_trigger": didaq_trigger, + "didaq_trigger_info_type": didaq_trigger_info_type, + "readout_time": readout_time, + "duration": duration + } + + + except KeyError as e: + logger.error(f"Key {e} not found in header file. Please check the structure of the header file.") + return None + +def assign_trigger_types(force_trigger, radiant_trigger, lt_trigger, which_trigger, default="UNKNOWN"): + if len(force_trigger) != len(radiant_trigger) or len(force_trigger) != len(lt_trigger) or len(force_trigger) != len(which_trigger): + logger.error("Trigger arrays must have the same length.") + return None + + n_events = len(force_trigger) + trigger_type_arr = np.full(n_events, default, dtype=' 1 + if np.any(overlap_mask): + overlap_idx = np.where(overlap_mask)[0] + logger.error(f"Found {len(overlap_idx)} events with overlapping trigger types at indices {overlap_idx}. Please check the trigger info.") + return None + + wrong_force = np.where(force_trigger & (trigger_type_arr != "FORCE"))[0] + if len(wrong_force) > 0: + logger.error(f"Found {len(wrong_force)} events where force_trigger is True but trigger type is not assigned as FORCE at indices {wrong_force}. Please check the trigger info.") + return None + + wrong_lt = np.where(lt_trigger & (trigger_type_arr != "LT"))[0] + if len(wrong_lt) > 0: + logger.error(f"Found {len(wrong_lt)} events where lt_trigger is True but trigger type is not assigned as LT at indices {wrong_lt}. Please check the trigger info.") + return None + + wrong_radiant = np.where(radiant_trigger & ~np.isin(trigger_type_arr, ["RADIANT0", "RADIANT1", "RADIANTX"]))[0] + if len(wrong_radiant) > 0: + logger.error(f"Found {len(wrong_radiant)} events where radiant_trigger is True but trigger type is not assigned as RADIANT at indices {wrong_radiant}. Please check the trigger info.") + return None + + unknown_idx = np.where(trigger_type_arr == default)[0] + if len(unknown_idx) > 0: + logger.warning(f"Found {len(unknown_idx)} events with trigger type not in FORCE, LT, RADIANT0, RADIANT1 or RADIANTX at indices {unknown_idx}. They are assigned as {default}.") + + return trigger_type_arr + +def assign_trigger_types_didaq(force_trigger, didaq_trigger, didaq_info_type, default="UNKNOWN"): + if len(force_trigger) != len(didaq_trigger) or len(force_trigger) != len(didaq_info_type): + logger.error("Trigger arrays must have the same length.") + return None + + n_events = len(force_trigger) + trigger_type_arr = np.full(n_events, default, dtype=' 1 + if np.any(overlap_mask): + overlap_idx = np.where(overlap_mask)[0] + logger.error(f"Found {len(overlap_idx)} events with overlapping trigger types at indices {overlap_idx}. Please check the trigger info.") + return None + + wrong_force = np.where(force_trigger & (trigger_type_arr != "FORCE"))[0] + if len(wrong_force) > 0: + logger.error(f"Found {len(wrong_force)} events where force_trigger is True but trigger type is not assigned as FORCE at indices {wrong_force}. Please check the trigger info.") + return None + + wrong_didaq = np.where(didaq_trigger & ~np.isin(trigger_type_arr, ["DIDAQ_COINC0", "DIDAQ_COINC1", "DIDAQ_DEEP_PHASED", "DIDAQ_SURF_UP", "DIDAQ_SURF_DOWN"]))[0] + if len(wrong_didaq) > 0: + logger.error(f"Found {len(wrong_didaq)} events where didaq_trigger is True but trigger type is not assigned as DIDAQ at indices {wrong_didaq}. Please check the trigger info.") + return None + + unknown_idx = np.where(trigger_type_arr == default)[0] + if len(unknown_idx) > 0: + logger.warning(f"Found {len(unknown_idx)} events with trigger type not in FORCE, DIDAQ_COINC0, DIDAQ_COINC1, DIDAQ_DEEP_PHASED, DIDAQ_SURF_UP or DIDAQ_SURF_DOWN at indices {unknown_idx}. They are assigned as {default}.") + + return trigger_type_arr + +def check_event_numbers_according_to_trigger_types_didaq(trigger_type_arr, n_forced_triggers,n_lt_triggers,n_rf0_triggers,n_rf1_triggers,): + + unique, counts = np.unique(trigger_type_arr, return_counts=True) + trigger_type_counts = dict(zip(unique, counts)) + + expected_counts = { + "FORCE": n_forced_triggers, + "DIDAQ_DEEP_PHASED": n_lt_triggers, + "DIDAQ_SURF_UP": n_rf0_triggers, + "DIDAQ_SURF_DOWN": n_rf1_triggers, + } + + for trigger_type, expected in expected_counts.items(): + found = trigger_type_counts.get(trigger_type, 0) + + if found != expected: + logger.error(f"Mismatch in trigger type counts for {trigger_type}: expected {expected}, found {found}. Please check the trigger info and event numbers.") + return False + + return True + +def check_event_numbers_according_to_trigger_types(trigger_type_arr, n_forced_triggers,n_lt_triggers,n_rf0_triggers,n_rf1_triggers,): + + unique, counts = np.unique(trigger_type_arr, return_counts=True) + trigger_type_counts = dict(zip(unique, counts)) + + expected_counts = { + "FORCE": n_forced_triggers, + "LT": n_lt_triggers, + "RADIANT0": n_rf0_triggers, + "RADIANT1": n_rf1_triggers, + } + + for trigger_type, expected in expected_counts.items(): + found = trigger_type_counts.get(trigger_type, 0) + + if found != expected: + logger.error(f"Mismatch in trigger type counts for {trigger_type}: expected {expected}, found {found}. Please check the trigger info and event numbers.") + return False + + return True + +def calculate_snr(max_abs_amplitude_arr, rms_arr): + snr_arr = np.full_like(max_abs_amplitude_arr, np.inf, dtype=float) + nonzero_mask = rms_arr != 0 + snr_arr[nonzero_mask] = max_abs_amplitude_arr[nonzero_mask] / rms_arr[nonzero_mask] + + if np.any(~nonzero_mask): + logger.warning(f"Found {np.sum(~nonzero_mask)} zero RMS values. Assigned SNR as np.inf.") + + return snr_arr + +def choose_trigger_type_header(trigger_type_arr, trigger_type:str, daq_type:str): + '''Choose events based on trigger type.''' + if daq_type == "didaq": + if trigger_type not in ["FORCE", "DIDAQ_DEEP_PHASED", "DIDAQ_SURF_UP", "DIDAQ_SURF_DOWN"]: + logger.error(f"Invalid trigger type {trigger_type}. Must be one of FORCE, DIDAQ_DEEP_PHASED, DIDAQ_SURF_UP or DIDAQ_SURF_DOWN.") + return None + elif daq_type == "radiant": + if trigger_type not in ["FORCE", "LT", "RADIANT0", "RADIANT1"]: + logger.error(f"Invalid trigger type {trigger_type}. Must be one of FORCE, LT, RADIANT0 or RADIANT1.") + return None + + mask = trigger_type_arr == trigger_type + return mask + +def read_multiple_runs(base_path, station_id, run_numbers, daq_type): + all_event_info = [] + + total_n_events = 0 + total_n_force_triggers = 0 + total_n_lt_triggers = 0 + total_n_rf0_triggers = 0 + total_n_rf1_triggers = 0 + + run_event_counts = {} + run_trigger_rates = {} + + spectrum_keys = ["avg_spectrum", "avg_spectrum_force", "avg_spectrum_lt", "avg_spectrum_rf0", "avg_spectrum_rf1"] # (n_ch, n_freqs) + + if daq_type == "didaq": + channel_event_keys = ["rms_arr", "max_abs_amplitude_arr", "snr_arr"] # (n_ch, n_events) + elif daq_type == "radiant": + channel_event_keys = ["rms_arr", "max_abs_amplitude_arr", "glitching_test_statistic_arr", "block_offsets_arr", "snr_arr"] # (n_ch, n_events) + event_keys = ["event_number_arr", "triggerType", "trigger_time_utc", "run_no", "station_id"] # 1D arrays with shape (n_events,) + + freqs = None + + failed_runs = [] + failed_run_info = {} + + for run_no in tqdm(run_numbers, desc=f"Reading monitoring and header files for runs between {run_numbers[0]} and {run_numbers[-1]} for station {station_id}"): + monitoring_file_path = os.path.join(base_path, f"station{station_id}/run{run_no}", "monitoring.root") + header_file_path = os.path.join(base_path, f"station{station_id}/run{run_no}", "headers.root") + + monitoring_file = open_file(monitoring_file_path) + header_file = open_file(header_file_path) + + # Skip run if either monitoring or the header file is missing and report + missing_files = [] + if monitoring_file is None: + missing_files.append(monitoring_file_path) + if header_file is None: + missing_files.append(header_file_path) + if missing_files: + msg = f"Missing files for run {run_no} for station {station_id}: {missing_files}. Skipping this run." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + event_info_dict = get_event_info_from_monitoring_file(monitoring_file, daq_type) + run_summary_dict = get_run_summary_from_monitoring_file(monitoring_file) + header_info_dict = get_info_from_header_file(header_file, daq_type) + + if event_info_dict is None: + msg = f"Failed to read event info for run {run_no} for station {station_id}. Skipping this run." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if run_summary_dict is None: + msg = f"Failed to read run summary info for run {run_no} for station {station_id}. Skipping this run." + logger.warning(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_info_dict is None: + msg = f"Failed to read header info for run {run_no} for station {station_id}. Skipping this run." + logger.warning(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + # Some sanity checks to make sure event number and run number are consistent between monitoring and header files + if not np.array_equal(event_info_dict["event_number_arr"], header_info_dict["event_number"]): + msg = f"Event numbers in monitoring file and header file do not match for run {run_no} for station {station_id}. Skipping the run. Please check the files." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + header_station_id = np.unique(header_info_dict["station_id"]) + header_run_no = np.unique(header_info_dict["run_no"]) + + run_summary_station_id = run_summary_dict["station_number"] + run_summary_run_no = run_summary_dict["run_number"] + + if header_station_id.size != 1: + msg = f"Multiple station IDs found in header file for run {run_no} for station {station_id}. Found station IDs: {header_station_id}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_run_no.size != 1: + msg = f"Multiple run numbers found in header file for run {run_no} for station {station_id}. Found run numbers: {header_run_no}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_station_id[0] != station_id: + msg = f"Station ID in header file ({header_station_id[0]}) does not match the station ID in the path ({station_id}) for run {run_no}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_station_id[0] != run_summary_station_id: + msg = f"Station ID in header file ({header_station_id[0]}) does not match the station ID in monitoring file run summary ({run_summary_station_id}) for run {run_no}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_run_no[0] != run_no: + msg = f"Run number in header file ({header_run_no[0]}) does not match the run number in the path ({run_no}) for run {run_no}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if header_run_no[0] != run_summary_run_no: + msg = f"Run number in header file ({header_run_no[0]}) does not match the run number in monitoring file run summary ({run_summary_run_no}) for run {run_no}. Skipping the run. Please check the file." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + # Start processing the run if all checks are passed + if daq_type == "radiant": + trigger_type_arr = assign_trigger_types( + header_info_dict["force_trigger"], + header_info_dict["radiant_trigger"], + header_info_dict["lt_trigger"], + header_info_dict["which_radiant"] + ) + elif daq_type == "didaq": + trigger_type_arr = assign_trigger_types_didaq( + header_info_dict["force_trigger"], + header_info_dict["didaq_trigger"], + header_info_dict["didaq_trigger_info_type"] + ) + + if trigger_type_arr is None: + msg = f"Failed to assign trigger types for run {run_no} for station {station_id}. Skipping this run." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + if daq_type == "radiant": + check_event_numbers = check_event_numbers_according_to_trigger_types( + trigger_type_arr, + run_summary_dict["n_forced_triggers"], + run_summary_dict["n_lt_triggers"], + run_summary_dict["n_rf0_triggers"], + run_summary_dict["n_rf1_triggers"], + ) + elif daq_type == "didaq": + check_event_numbers = check_event_numbers_according_to_trigger_types_didaq( + trigger_type_arr, + run_summary_dict["n_forced_triggers"], + run_summary_dict["n_didaq_deep_phased_triggers"], + run_summary_dict["n_didaq_surf_up_triggers"], + run_summary_dict["n_didaq_surf_down_triggers"], + ) + + if not check_event_numbers: + msg = f"Event number check according to trigger types failed for run {run_no} for station {station_id}. Skipping this run." + logger.error(msg) + failed_runs.append(run_no) + failed_run_info[run_no] = msg + continue + + # Add trigger type and some header info to event info dict for easier access later for each event + event_info_dict["triggerType"] = trigger_type_arr # (n_events,) + event_info_dict["trigger_time_utc"] = header_info_dict["trigger_time_utc"] # (n_events,) + event_info_dict["run_no"] = header_info_dict["run_no"] # (n_events,) + event_info_dict["station_id"] = header_info_dict["station_id"] # (n_events,) + event_info_dict["duration"] = header_info_dict["duration"] # a number representing the duration of the run in seconds, calculated as the difference between the max and min readout time in the header file + event_info_dict["readout_time"] = header_info_dict["readout_time"] # (n_events,) + + # Add runsummary + if daq_type == "radiant": + event_info_dict["avg_spectrum"] = stack_if_object(run_summary_dict["avg_spectrum"]) # (n_ch, n_freqs) + event_info_dict["avg_spectrum_force"] = stack_if_object(run_summary_dict["avg_spectrum_force"]) # (n_ch, n_freqs) + event_info_dict["avg_spectrum_lt"] = stack_if_object(run_summary_dict["avg_spectrum_lt"]) # (n_ch, n_freqs) + event_info_dict["avg_spectrum_rf0"] = stack_if_object(run_summary_dict["avg_spectrum_rf0"]) #RADIANT0 + event_info_dict["avg_spectrum_rf1"] = stack_if_object(run_summary_dict["avg_spectrum_rf1"]) #RADIANT1 + + elif daq_type == "didaq": + event_info_dict["avg_spectrum"] = stack_if_object(run_summary_dict["avg_spectrum"]) # (n_ch, n_freqs) + event_info_dict["avg_spectrum_force"] = stack_if_object(run_summary_dict["avg_spectrum_force"]) # (n_ch, n_freqs) + event_info_dict["avg_spectrum_lt"] = stack_if_object(run_summary_dict["avg_spectrum_didaq_deep_phased"]) #DIDAQ_DEEP_PHASED + event_info_dict["avg_spectrum_rf0"] = stack_if_object(run_summary_dict["avg_spectrum_didaq_surf_up"]) #DIDAQ_SURF_UP + event_info_dict["avg_spectrum_rf1"] = stack_if_object(run_summary_dict["avg_spectrum_didaq_surf_down"]) #DIDAQ_SURF_DOWN + + # Calculate SNR and add to event info dict + snr_arr = calculate_snr(event_info_dict["max_abs_amplitude_arr"], event_info_dict["rms_arr"]) + event_info_dict["snr_arr"] = snr_arr # (n_ch, n_events) + + # Count total events: + if daq_type == "radiant": + total_n_events += run_summary_dict["n_events"] + total_n_force_triggers += run_summary_dict["n_forced_triggers"] + total_n_lt_triggers += run_summary_dict["n_lt_triggers"] + total_n_rf0_triggers += run_summary_dict["n_rf0_triggers"] + total_n_rf1_triggers += run_summary_dict["n_rf1_triggers"] + + elif daq_type == "didaq": + total_n_events += run_summary_dict["n_events"] + total_n_force_triggers += run_summary_dict["n_forced_triggers"] + total_n_lt_triggers += run_summary_dict["n_didaq_deep_phased_triggers"] + total_n_rf0_triggers += run_summary_dict["n_didaq_surf_up_triggers"] + total_n_rf1_triggers += run_summary_dict["n_didaq_surf_down_triggers"] + + # Calculate trigger rates and add to run_trigger_rates dict for this run + if daq_type == "radiant": + trigger_rate_force = run_summary_dict["n_forced_triggers"] / event_info_dict["duration"] + trigger_rate_lt = run_summary_dict["n_lt_triggers"] / event_info_dict["duration"] + trigger_rate_rf0 = run_summary_dict["n_rf0_triggers"] / event_info_dict["duration"] + trigger_rate_rf1 = run_summary_dict["n_rf1_triggers"] / event_info_dict["duration"] + + elif daq_type == "didaq": + trigger_rate_force = run_summary_dict["n_forced_triggers"] / event_info_dict["duration"] + trigger_rate_lt = run_summary_dict["n_didaq_deep_phased_triggers"] / event_info_dict["duration"] + trigger_rate_rf0 = run_summary_dict["n_didaq_surf_up_triggers"] / event_info_dict["duration"] + trigger_rate_rf1 = run_summary_dict["n_didaq_surf_down_triggers"] / event_info_dict["duration"] + + run_trigger_rates[run_no] = { + "force_trigger_rate": trigger_rate_force, + "lt_trigger_rate": trigger_rate_lt, + "rf0_trigger_rate": trigger_rate_rf0, + "rf1_trigger_rate": trigger_rate_rf1, + "run_start_time_utc": np.min(event_info_dict["readout_time"].astype("datetime64[s]")), + } + + # Add event counts for this run to the run_event_counts dict + if daq_type == "radiant": + run_event_counts[run_no] = { + "n_events": run_summary_dict["n_events"], + "n_forced_triggers": run_summary_dict["n_forced_triggers"], + "n_lt_triggers": run_summary_dict["n_lt_triggers"], + "n_rf0_triggers": run_summary_dict["n_rf0_triggers"], + "n_rf1_triggers": run_summary_dict["n_rf1_triggers"], + } + elif daq_type == "didaq": + run_event_counts[run_no] = { + "n_events": run_summary_dict["n_events"], + "n_forced_triggers": run_summary_dict["n_forced_triggers"], + "n_lt_triggers": run_summary_dict["n_didaq_deep_phased_triggers"], + "n_rf0_triggers": run_summary_dict["n_didaq_surf_up_triggers"], + "n_rf1_triggers": run_summary_dict["n_didaq_surf_down_triggers"], + } + + + if freqs is None: + freqs = stack_if_object(run_summary_dict["frequencies"]) # (n_freqs,) + + all_event_info.append(event_info_dict) + + if len(all_event_info) == 0: + raise ValueError(f"No valid runs were processed for station {station_id}. Please check the files and the run numbers. Or try reading using the dataProviderRNOG method which reads from combined.root files in /inbox/ and can be used for older data before 2026 which do not have monitoring.root files.") + + combined_event_info = {} + for key in channel_event_keys: + combined_event_info[key] = np.concatenate([event_info[key] for event_info in all_event_info], axis=1) # concatenate along events axis, so final shape is (n_ch, n_events_total) + for key in event_keys: + combined_event_info[key] = np.concatenate([event_info[key] for event_info in all_event_info], axis=0) # concatenate along events axis, so final shape is (n_events_total,) + for key in spectrum_keys: + combined_event_info[key] = np.stack([event_info[key] for event_info in all_event_info], axis=1) # stack along new axis for runs, so final shape is (n_ch, n_runs, n_freqs) + + combined_event_info["freqs"] = freqs # (n_freqs,) + combined_event_info["total_n_events"] = total_n_events + combined_event_info["total_n_force_triggers"] = total_n_force_triggers + combined_event_info["total_n_lt_triggers"] = total_n_lt_triggers + combined_event_info["total_n_rf0_triggers"] = total_n_rf0_triggers + combined_event_info["total_n_rf1_triggers"] = total_n_rf1_triggers + combined_event_info["run_event_counts"] = run_event_counts # dict with run number as key and value as another dict with n_events, n_forced_triggers, n_lt_triggers, n_rf0_triggers, n_rf1_triggers for that run + combined_event_info["run_trigger_rates"] = run_trigger_rates # dict with run number as key and value as another dict with force_trigger_rate, lt_trigger_rate, rf0_trigger_rate, rf1_trigger_rate for that run + combined_event_info["failed_runs"] = failed_runs if len(failed_runs) > 0 else None + combined_event_info["failed_run_info"] = failed_run_info if len(failed_run_info) > 0 else None + + logger.info(f"Successfully read and combined data from {len(all_event_info)} runs for station {station_id} using the monitoring data. Total events: {total_n_events}, total FORCE triggers: {total_n_force_triggers}, total LT triggers: {total_n_lt_triggers}, total RADIANT0 triggers: {total_n_rf0_triggers}, total RADIANT1 triggers: {total_n_rf1_triggers}.") + + return combined_event_info diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/config_station.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/config_station.py new file mode 100644 index 0000000..c4fd915 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/config_station.py @@ -0,0 +1,35 @@ +from NuRadioReco.utilities import units + + +DEFAULT_CONFIG = { + "all_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 , 19 , 20, 21, 22, 23], + "surface_channels": [12, 13, 14, 15, 16, 17, 18 , 19 , 20], + "deep_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 21, 22, 23], + "upward_channels": [13, 16, 19], + "downward_channels": [12, 14, 15, 17, 18, 20], + "vpol_channels": [0, 1, 2, 3, 5, 6, 7, 9, 10, 22, 23], + "hpol_channels": [4, 8, 11, 21], + "phased_array_channels": [0, 1, 2, 3], + "reference_channels": [12, 14, 15, 17, 18, 20], # downward facing surface channels + "reference_channels_galaxy": [12, 14, 15, 17, 18, 20], # downward facing surface channels +} + +STATION_SPECIFIC_ADJUSTMENTS = { + 14: { + "surface_channels": [12, 13, 14, 15, 16, 17, 18 , 19], + "deep_channels": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23], + "upward_channels": [13, 15, 16, 18], + "downward_channels": [12, 14, 17, 19], + "vpol_channels": [0, 1, 2, 3, 5, 6, 7, 9, 10, 20, 22, 23], # 20 is a vpol channel in station 14 instead of lpda + "reference_channels": [12, 14, 17, 19], # downward facing surface channels for station 14 + "reference_channels_galaxy": [12, 14, 19], # downward facing surface channels for station 14 except for 17, which behaves weirdly in that region + } +} + +def get_station_config(station_id): + cfg = DEFAULT_CONFIG.copy() + if station_id in STATION_SPECIFIC_ADJUSTMENTS: + adjustments = STATION_SPECIFIC_ADJUSTMENTS[station_id] + cfg.update(adjustments) + return cfg + diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/trigger_rate.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/trigger_rate.py new file mode 100644 index 0000000..a107941 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/outdated/trigger_rate.py @@ -0,0 +1,99 @@ +import os +import datetime +import numpy as np +from matplotlib import pyplot as plt +import pandas as pd +import matplotlib.dates as mdates +from datetime import timezone + + +#### Trigger analysis (adapted from the plot_trigger() function from analyze_run.py) #### +def compute_radiant_thresholds(event_info, down_channels, up_channels): + radiant = event_info["radiantThrs"] + + downward = radiant[:, down_channels].mean(axis=1) + upward = radiant[:, up_channels].mean(axis=1) + low_trig = event_info["lowTrigThrs"].mean(axis=1) + + return upward, downward, low_trig + +def plot_trigger_rate_with_thresholds(station_id, event_info, down_channels, up_channels, run_label, day_interval, bin_width_initial=300, max_bins=800, save_location=None): + + trigger_times = np.asarray(event_info["triggerTime"]) + readout_times = np.asarray(event_info["readoutTime"]) + + run_duration = trigger_times.max() - trigger_times.min() + run_duration_readout = readout_times.max() - readout_times.min() + + bin_width = bin_width_initial + nbins = int(run_duration // bin_width) + if nbins > max_bins: + bin_width = 3600 # 1 hour + nbins = int(run_duration // bin_width) + + times = np.array([datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc) for ts in trigger_times]) + time_span = times.max() - times.min() + time_span_days = time_span.total_seconds() / 86400.0 # convert to days + + fig, ax_rate = plt.subplots(figsize=(12, 6)) + + ax_rate.grid(True, which="both", ls="--", lw=0.35, alpha=0.5) + + weights_total = np.full(times.shape[0], 1.0 / bin_width) + _, bin_edges, _ = ax_rate.hist(times, bins=nbins, weights=weights_total, histtype="step", color="k", label="Total Rate",) + + triggers = np.unique(event_info["triggerType"]) + trigger_colors = { + "FORCE": "tab:blue", + "RADIANT0": "tab:orange", + "RADIANT1": "tab:green", + "LT": "tab:red",} + + for trigger in triggers: + mask = event_info["triggerType"] == trigger + n_mask = mask.sum() + if n_mask == 0: + continue + + color = trigger_colors.get(trigger) + + ax_rate.hist(times[mask], bins=bin_edges, weights=np.full(n_mask, 1.0 / bin_width), histtype="step", lw=1.1, label=str(trigger), color=color,) + + ax_rate.set_ylabel("Trigger Rate [Hz]") + ax_rate.set_yscale("log") + + if time_span_days < 1: + # Use 6h ticks if less than 1 day + ax_rate.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ax_rate.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span_days < 3: + # Use 12h ticks if less than 3 days + ax_rate.xaxis.set_major_locator(mdates.HourLocator(interval=12)) + ax_rate.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + # Use day ticks otherwise + ax_rate.xaxis.set_major_locator(mdates.DayLocator(interval = day_interval)) + ax_rate.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz = timezone.utc)) + + ax_rate.tick_params(axis="x", rotation=25) + ax_rate.set_xlabel("Time (UTC)") + + upward, downward, lt = compute_radiant_thresholds(event_info, down_channels, up_channels) + scale = 2.5 / 16777215.0 # convert register to Volts + + ax_thr = ax_rate.twinx() + + ax_thr.plot(times, upward * scale, ls="--", lw=2, color="darkmagenta", label="RADIANT Up (avg)",) + ax_thr.plot(times, downward * scale, ls="--", lw=2, color="darkgreen", label="RADIANT Down (avg)",) + ax_thr.plot(times, lt * scale, ls="--", lw=2, color="mediumblue", label="LT (avg)",) + + ax_thr.set_ylabel("Threshold [V]") + + h1, l1 = ax_rate.get_legend_handles_labels() + h2, l2 = ax_thr.get_legend_handles_labels() + ax_rate.legend(h1 + h2, l1 + l2, loc="upper left", bbox_to_anchor=(1.1, 1), borderaxespad=0., frameon=True, framealpha=1.0,) + + fig.tight_layout() + fig.savefig(os.path.join(save_location, f"trigger_rate_with_thresholds_{station_id}_{run_label}.pdf")) + + return fig, ax_rate, ax_thr \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/__init__.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_debug.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_debug.py new file mode 100644 index 0000000..4569dfc --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_debug.py @@ -0,0 +1,290 @@ +import matplotlib.pyplot as plt +import numpy as np +import os + +#### Debug Plots #### +def debug_plot_ratios(ratio_arr_dict, channels_order, save_location, station_id, run_label, bins=30): + n_bands = len(ratio_arr_dict) + + fig, axes = plt.subplots(1, n_bands, figsize=(5*n_bands, 5), sharey=True) + + if n_bands == 1: + axes = [axes] + + for ax, (band_name, ratio_list) in zip(axes, ratio_arr_dict.items()): + for ch, r in zip(channels_order, ratio_list): + r = np.asarray(r) + ax.hist(np.log10(r), bins=bins, histtype="step", linewidth=1.3, label=f"Ch {ch}", alpha=0.8) + + ax.set_title(band_name) + ax.set_xlabel("log10(R)") + ax.grid(True, alpha=0.3) + + ax.set_title(f"{band_name} (FORCE Trigger)") + ax.set_xlabel("log10(R)") + ax.grid(True, alpha=0.3) + + axes[0].set_ylabel("Counts") + + handles, labels = axes[0].get_legend_handles_labels() + fig.legend(handles, labels, loc="upper right", ncol=8, frameon=True) + plt.tight_layout() + fig.savefig(os.path.join(save_location,f"debug_amplitude_ratios_force_trigger_{station_id}_{run_label}.pdf",)) + plt.close(fig) + +def debug_plot_snr_distribution(log_snr_arr, channel_list, save_location, station_id, run_label, bins=30): + fig, ax = plt.subplots(figsize=(10, 6)) + for ch in channel_list: + log_snr_ch = log_snr_arr[ch] + ax.hist(log_snr_ch, bins=bins, histtype="step", linewidth=1.3, label=f"Ch {ch}", alpha=0.8) + + #ax.set_title("Log10 SNR Distribution (FORCE Trigger)") + ax.set_xlabel("log10(SNR)") + ax.set_ylabel("Counts") + ax.grid(True, alpha=0.3) + + handles, labels = ax.get_legend_handles_labels() + fig.legend(handles, labels, loc="upper right", ncol=8, frameon=True) + plt.tight_layout() + fig.savefig(os.path.join(save_location,f"debug_snr_distribution_force_trigger_{station_id}_{run_label}.pdf",)) + plt.close(fig) + +def debug_plot_z_score_snr(z_score_arr, channel_list, save_location, station_id, run_label, bins=30): + fig, ax = plt.subplots(figsize=(10, 6)) + for ch in channel_list: + z_score_ch = z_score_arr[ch] + ax.hist(z_score_ch, bins=bins, histtype="step", linewidth=1.3, label=f"Ch {ch}", alpha=0.8) + + #ax.set_title("Z-Score SNR Distribution (FORCE Trigger)") + ax.set_xlabel("Z-Score(SNR)") + ax.set_ylabel("Counts") + ax.grid(True, alpha=0.3) + + handles, labels = ax.get_legend_handles_labels() + fig.legend(handles, labels, loc="upper right", ncol=8, frameon=True) + plt.tight_layout() + fig.savefig(os.path.join(save_location,f"debug_z_score_snr_force_trigger_{station_id}_{run_label}.pdf",)) + plt.close(fig) + +def debug_plot_vrms_distribution(vrms_arr, modality_dict, channel_list, station_id, run_label, trigger_label, save_location, n_rows=12, n_cols=2, use_monitoring=False): + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "RMS" + else: + unit_label = "Vrms Values [V]" + plot_label = "Vrms" + + fig, axes = plt.subplots(n_rows, n_cols, figsize=(16, 36)) + axes = axes.flatten() + + for idx, ch in enumerate(channel_list): + ax = axes[idx] + info = modality_dict[ch] + vrms = vrms_arr[ch] + vrms = vrms[np.isfinite(vrms)] + vrms_grid = info["vrms_grid"] + kde_values = info["kde_values"] + peaks = info["peaks"] + n_peaks = info["n_peaks"] + + if n_peaks == 0: + modality = "flat/noisy" + elif n_peaks == 1: + modality = "unimodal" + elif n_peaks == 2: + modality = "bimodal" + else: + modality = f"multimodal ({n_peaks})" + + # histogram + ax.hist(vrms, bins=30, density=True, alpha=0.3, color="gray") + # kde curve + ax.plot(vrms_grid, kde_values, color="blue", lw=1.5) + # peaks + if len(peaks) > 0: + ax.plot(vrms_grid[peaks], kde_values[peaks], "ro", markersize=5) + + ax.set_title(f"Ch {ch}: {modality}") + ax.set_xlabel(unit_label) + ax.set_ylabel("KDE Density") + + for i in range(len(channel_list), n_rows * n_cols): + axes[i].axis("off") + + plt.tight_layout() + plt.savefig(os.path.join(save_location,f"debug_{plot_label.lower()}_hist_kde_density_peaks_{station_id}_{run_label}_{trigger_label}.pdf",)) + +def debug_plot_ratios_just_galaxy( + ratio_arr_dict, + channels, + save_location, + station_id, + run_label, + bins=30, +): + """ + Plot galactic-excess ratio distributions for selected surface channels. + + The channel axis in ratio_arr_dict["galactic_excess"] is assumed to follow + the order given in surface_channels, rather than the numerical channel index. + """ + band_name = "galactic_excess" + surface_channels = [13, 15, 16, 18, 12, 14, 17, 19] + + if band_name not in ratio_arr_dict: + raise KeyError( + f"'{band_name}' not found in ratio_arr_dict. " + f"Available keys: {list(ratio_arr_dict.keys())}" + ) + + ratio_list = ratio_arr_dict[band_name] + + if len(ratio_list) != len(surface_channels): + raise ValueError( + f"Expected {len(surface_channels)} channel entries in " + f"ratio_arr_dict['{band_name}'], but found {len(ratio_list)}." + ) + + channel_to_index = { + ch: index for index, ch in enumerate(surface_channels) + } + + fig, ax = plt.subplots(figsize=(6, 5)) + + for ch in channels: + if ch not in channel_to_index: + print( + f"Warning: channel {ch} is not available. " + f"Available channels: {surface_channels}" + ) + continue + + if ch in [13, 15, 16, 18]: + up_label = "Up" + elif ch in [12, 14, 17, 19]: + up_label = "Down" + else: + up_label = "unknown" + + channel_index = channel_to_index[ch] + ratios = np.asarray(ratio_list[channel_index], dtype=float) + + # log10 is only defined for finite, positive values. + valid_mask = np.isfinite(ratios) & (ratios > 0) + ratios = ratios[valid_mask] + + if ratios.size == 0: + print( + f"Warning: channel {ch} contains no finite, positive ratios." + ) + continue + + ax.hist( + np.log10(ratios), + bins=bins, + histtype="step", + linewidth=1.3, + alpha=0.8, + label=f"Ch {ch} ({up_label})", + ) + + #ax.set_title("Galactic Excess (FORCE Trigger)") + ax.set_xlabel(r"$\log_{10}(R)$") + ax.set_ylabel("Counts") + ax.grid(True, alpha=0.3) + + if ax.has_data(): + ax.legend(frameon=True) + + fig.tight_layout() + + output_path = os.path.join( + save_location, + ( + f"debug_amplitude_ratios_force_trigger_" + f"{station_id}_{run_label}_just_galaxy.pdf" + ), + ) + + fig.savefig(output_path) + plt.close(fig) + +def debug_plot_vrms_distribution_single_channel( + vrms_arr, + modality_dict, + channel, + station_id, + run_label, + trigger_label, + save_location, + use_monitoring=False +): + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "RMS" + else: + unit_label = "Vrms Values [V]" + plot_label = "Vrms" + + fig, ax = plt.subplots(figsize=(8,6)) + + info = modality_dict[channel] + + vrms = vrms_arr[channel] + vrms = vrms[np.isfinite(vrms)] + + vrms_grid = info["vrms_grid"] + kde_values = info["kde_values"] + peaks = info["peaks"] + n_peaks = info["n_peaks"] + + if n_peaks == 0: + modality = "flat/noisy" + elif n_peaks == 1: + modality = "unimodal" + elif n_peaks == 2: + modality = "bimodal" + else: + modality = f"multimodal ({n_peaks})" + + ax.hist( + vrms, + bins=30, + density=True, + alpha=0.3, + color="gray", + label=f"{plot_label} distribution", + ) + + ax.plot( + vrms_grid, + kde_values, + color="blue", + lw=1.5, + label="Gaussian KDE", + ) + + if len(peaks) > 0: + ax.plot( + vrms_grid[peaks], + kde_values[peaks], + "ro", + markersize=5, + label="Peak", + ) + + #ax.set_title(f"Ch {channel}: {modality}") + ax.set_xlabel(unit_label) + ax.set_ylabel("Density") + ax.legend() + + plt.tight_layout() + + plt.savefig( + os.path.join( + save_location, + f"debug_{plot_label.lower()}_hist_kde_density_peaks_ch{channel}_{station_id}_{run_label}_{trigger_label}.pdf" + ) + ) + + plt.close(fig) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_glitch.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_glitch.py new file mode 100644 index 0000000..826de79 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_glitch.py @@ -0,0 +1,80 @@ +import matplotlib.pyplot as plt +from matplotlib import colors, cm +import numpy as np +import os +import pandas as pd + +#### Glitching Plots #### +def glitching_violin_plot(glitch_arr, channel_list, station_id, run_label, save_location): + data = [glitch_arr[ch] for ch in channel_list] + means = np.array([np.mean(glitch_arr[ch]) for ch in channel_list]) + + fig, ax = plt.subplots(figsize=(12, 10)) + + parts = ax.violinplot(data, positions=channel_list, showextrema=True, showmedians=True, vert=False, side="high", widths=1.8,) + + norm = colors.Normalize(vmin=np.min(means), vmax=np.max(means), clip=False) + sm = cm.ScalarMappable(norm=norm, cmap="Blues") + + for idx, pc in enumerate(parts["bodies"]): + pc.set_facecolor(sm.to_rgba(means[idx])) + pc.set_alpha(0.5) + pc.set_edgecolor("k") + + parts["cmins"].set_linewidth(0.2) + parts["cmaxes"].set_linewidth(0.2) + parts["cbars"].set_linewidth(0.5) + parts["cmins"].set_color("k") + parts["cmaxes"].set_color("k") + parts["cbars"].set_color("k") + parts["cmedians"].set_color("k") + parts["cmedians"].set_linewidth(1) + + cb = plt.colorbar(sm, ax=ax, pad=0.02) + cb.set_label("Mean test statistics") + + ax.set_xlabel("Glitching test statistics") + ax.set_ylabel("Channel") + ax.set_yticks(channel_list) + ax.grid(True, alpha=0.3) + ax.set_ylim(min(channel_list) - 1, max(channel_list) + 1) + + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"glitching_violin_plot_{station_id}_{run_label}.pdf")) + plt.close(fig) + +def choose_bin_size(times): + total_hours = (times.max() - times.min()).total_seconds() / 3600.0 + if total_hours < 12: + return "30min" + elif total_hours < 24: + return "1h" + elif total_hours < 3 * 24: + return "2h" + elif total_hours < 7 * 24: + return "6h" + elif total_hours < 30 * 24: + return "12h" + else: + return "24h" + +def plot_glitch_q99_over_time(times, glitch_arr, channels, station_id, run_label, save_location): + times = pd.to_datetime(times) + df = pd.DataFrame(glitch_arr.T, index=times, columns=channels) + + bin_rule = choose_bin_size(times) + q99 = df.resample(bin_rule).quantile(0.99) + + fig, ax = plt.subplots(figsize=(12, 10)) + + for ch in channels: + ax.plot(q99.index, q99[ch], marker=".", linestyle="-", label=f"ch {ch}") + + ax.set_xlabel("Date [UTC]") + ax.set_ylabel(f"99% quantile glitching ts ({bin_rule} bins)") + ax.grid(True, alpha=0.3) + ax.legend(ncol=3, frameon=True, framealpha=0.9, edgecolor="black") + + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"glitch_q99_{station_id}_{run_label}.pdf")) + plt.close(fig) diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_snr.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_snr.py new file mode 100644 index 0000000..3e33abd --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_snr.py @@ -0,0 +1,259 @@ +import matplotlib.pyplot as plt +import matplotlib.dates as mdates +from matplotlib.lines import Line2D +import numpy as np +import pandas as pd +import os +from datetime import timezone + +#### SNR Plots #### + +def choose_day_interval(times): + times = pd.to_datetime(times, utc=True) + total_days = (times.max() - times.min()).days + + if total_days < 10: + return 1 + elif total_days < 20: + return 2 + elif total_days < 40: + return 4 + elif total_days < 80: + return 7 + elif total_days < 150: + return 10 + elif total_days < 300: + return 15 + elif total_days < 600: + return 30 + else: + return 60 + +def plot_snr_against_time(station_id,times,snr_arr,flag,z_log,k_list,channels,save_location,run_label,nrows=12,ncols=2,day_interval=None): + times = pd.to_datetime(times,utc=True) + times = times.tz_convert(None) + + channels = list(channels) + n_channels = len(channels) + + if day_interval is None: + day_interval = choose_day_interval(times) + + fig, axs = plt.subplots(nrows,ncols,figsize=(15,24),sharex=True) + axs = np.array(axs) + + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + for idx, ch in enumerate(channels): + r = idx//ncols + c = idx%ncols + ax = axs[r,c] + + good_mask = ~flag[ch] + plot_mask = good_mask & ~pd.isna(times) + + ax.scatter(times[plot_mask], np.log10(snr_arr[ch][plot_mask]), s=8,alpha=0.25, color="gray", rasterized=True) + + zex = np.abs(z_log[ch]) - k_list[ch] + zex = np.clip(zex,0,None) + + sc = ax.scatter(times[flag[ch]], np.log10(snr_arr[ch][flag[ch]]), s=8,c=zex[flag[ch]], cmap="Reds", rasterized=True) + + cax = ax.inset_axes([1.02,0.1,0.05,0.8]) + plt.colorbar(sc,cax=cax, label=r"$|z|-k$") + + if time_span < 1: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span < 3: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + ax.set_xlabel("Date [UTC]") + ax.set_ylabel(r"$\log_{10}(\mathrm{SNR})$") + ax.tick_params(axis="x", labelbottom=True) + ax.grid(alpha=0.4) + ax.text(0.85, 0.95, f"Ch {ch}", transform=ax.transAxes, ha="left", va="top", bbox=dict(boxstyle="round, pad=0.25", facecolor="white", alpha=0.8)) + + for idx in range(n_channels, nrows*ncols): + r = idx//ncols + c = idx%ncols + axs[r,c].set_visible(False) + + red = plt.cm.Reds(0.6) + + legend_handles = [ + Line2D([0],[0],marker="o",color="none",markeredgecolor="gray",markerfacecolor="gray",markersize=6,label=r"$|z|\leq k$"), + Line2D([0],[0],marker="o",color="none",markeredgecolor=red,markerfacecolor=red,markersize=6,label=r"$|z|>k$") + ] + + axs[0,0].legend(handles=legend_handles, loc="upper left") + + fig.autofmt_xdate() + + plt.subplots_adjust(bottom=0.07, wspace=0.38, hspace=0.45, left=0.08) + plt.savefig(os.path.join(save_location,f"snr_against_time_{station_id}_{run_label}.pdf")) + plt.close(fig) + + +def plot_snr_against_time_per_trigger(station_id,times,snr_arr,channels,save_location,run_label,nrows=12,ncols=2,day_interval=None,color="blue",triggerlabel=""): + times = pd.to_datetime(times,utc=True) + times = times.tz_convert(None) + + channels = list(channels) + n_channels = len(channels) + + if day_interval is None: + day_interval = choose_day_interval(times) + + fig, axs = plt.subplots(nrows,ncols,figsize=(15,24),sharex=True) + axs = np.array(axs) + + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + for idx, ch in enumerate(channels): + r = idx//ncols + c = idx%ncols + ax = axs[r,c] + + ax.scatter(times, np.log10(snr_arr[ch]), s=8,alpha=0.25, color=color, rasterized=True) + + if time_span < 1: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span < 3: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + ax.set_xlabel("Date [UTC]") + ax.set_ylabel(r"$\log_{10}(\mathrm{SNR})$") + ax.tick_params(axis="x", labelbottom=True) + ax.grid(alpha=0.4) + ax.text(0.85, 0.95, f"Ch {ch}", transform=ax.transAxes, ha="left", va="top", bbox=dict(boxstyle="round, pad=0.25", facecolor="white", alpha=0.8)) + + for idx in range(n_channels, nrows*ncols): + r = idx//ncols + c = idx%ncols + axs[r,c].set_visible(False) + + legend_handles = [ + Line2D([0],[0],marker="o",color="none",markeredgecolor=color,markerfacecolor=color,markersize=6,label=r"SNR values") + ] + + axs[0,0].legend(handles=legend_handles, loc="upper left") + + fig.autofmt_xdate() + + plt.subplots_adjust(bottom=0.07, wspace=0.38, hspace=0.45, left=0.08) + plt.savefig(os.path.join(save_location,f"snr_against_time_{station_id}_{run_label}_{triggerlabel}.pdf")) + plt.close(fig) + +def plot_snr_against_time_single_channel( + station_id, + times, + snr_arr, + channel, + save_location, + run_label, + flag=None, + z_log=None, + k_list=None, + day_interval=None, + triggerlabel="" +): + times = pd.to_datetime(times, utc=True) + times = times.tz_convert(None) + + if day_interval is None: + day_interval = choose_day_interval(times) + + fig, ax = plt.subplots(figsize=(10,5)) + + if flag is None: + ax.scatter(times, np.log10(snr_arr[channel]), s=8, alpha=0.25, rasterized=True) + + legend_handles = [ + Line2D([0],[0],marker="o",color="none",markersize=6,label="SNR values") + ] + else: + good_mask = ~flag[channel] + plot_mask = good_mask & ~pd.isna(times) + + ax.scatter( + times[plot_mask], + np.log10(snr_arr[channel][plot_mask]), + s=8, + alpha=0.25, + color="gray", + rasterized=True + ) + + zex = np.abs(z_log[channel]) - k_list[channel] + zex = np.clip(zex,0,None) + + sc = ax.scatter( + times[flag[channel]], + np.log10(snr_arr[channel][flag[channel]]), + s=8, + c=zex[flag[channel]], + cmap="Reds", + rasterized=True + ) + + plt.colorbar(sc, ax=ax, label=r"$|z|-k$") + + red = plt.cm.Reds(0.6) + + legend_handles = [ + Line2D([0],[0],marker="o",color="none",markeredgecolor="gray",markerfacecolor="gray",markersize=6,label=r"$|z|\leq k$"), + Line2D([0],[0],marker="o",color="none",markeredgecolor=red,markerfacecolor=red,markersize=6,label=r"$|z|>k$") + ] + + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + if time_span < 1: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span < 3: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + ax.set_xlabel("Date [UTC]") + ax.set_ylabel(r"$\log_{10}(\mathrm{SNR})$") + ax.grid(alpha=0.4) + + ax.text( + 0.85, + 0.95, + f"Ch {channel}", + transform=ax.transAxes, + ha="left", + va="top", + bbox=dict(boxstyle="round, pad=0.25", facecolor="white", alpha=0.8) + ) + + ax.legend(handles=legend_handles, loc="upper left") + + fig.autofmt_xdate() + + plt.tight_layout() + + suffix = f"_{triggerlabel}" if triggerlabel else "" + + plt.savefig( + os.path.join( + save_location, + f"snr_against_time_ch{channel}_{station_id}_{run_label}{suffix}.pdf" + ) + ) + + plt.close(fig) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_spectrum.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_spectrum.py new file mode 100644 index 0000000..b907cd0 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_spectrum.py @@ -0,0 +1,246 @@ +import matplotlib.pyplot as plt +from matplotlib.lines import Line2D +from matplotlib.patches import Patch +import matplotlib.dates as mdates +import numpy as np +from NuRadioReco.utilities import units +import os +import logging + +logger = logging.getLogger(__name__) + +TRIGGER_MAP = { + "force" : "n_forced_triggers", + "lt" : "n_lt_triggers", + "radiant0" : "n_rf0_triggers", + "radiant1" : "n_rf1_triggers", +} + +TRIGGER_MAP_DIDAQ = { + "force" : "n_forced_triggers", + "didaq_deep_phased" : "n_lt_triggers", + "didaq_surf_up" : "n_rf0_triggers", + "didaq_surf_down" : "n_rf1_triggers", +} + + +def get_weights_if_monitoring(trigger_label, use_monitoring=False, run_event_counts=None, daq_type="radiant"): + '''Helper function to get weights for averaging spectra if using monitoring data, otherwise return None.''' + if use_monitoring and run_event_counts is not None: + trigger_map = TRIGGER_MAP_DIDAQ if daq_type == "didaq" else TRIGGER_MAP + weight_key = trigger_map[trigger_label.lower()] + n_events_per_run_trigger = np.array([run_event_counts[run_no][weight_key] for run_no in run_event_counts]) + unit_label = "ADC Counts" + return n_events_per_run_trigger, unit_label + else: + unit_label = "V/GHz" + return None, unit_label + +#### Spectrum Plots #### +def plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr, freqs, upward_channels, downward_channels, save_location, run_label, trigger_label, use_monitoring = False, run_event_counts = None, daq_type="radiant"): + '''Plot time-integrated surface channel spectra. Use weighted average if use_monitoring is True and run_event_counts is provided, otherwise use simple average.''' + + plt.figure(figsize=(10, 6)) + weights, unit_label = get_weights_if_monitoring(trigger_label, use_monitoring, run_event_counts, daq_type) + #print(f"shape of spec_arr: {spec_arr.shape}, shape of freqs: {freqs.shape}") + for ch in upward_channels: + if weights is not None: + spec_mean = np.average(spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (up)', linestyle='-') + for ch in downward_channels: + if weights is not None: + spec_mean = np.average(spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (down)', linestyle='--') + + plt.xlabel('Frequency [MHz]') + plt.xlim(50, 800) + #plt.ylim(0, 5) + plt.ylabel(f'Amplitude Spectrum [{unit_label}]') + plt.title(f'Time-Integrated Spectrum of Surface Channels ({trigger_label} Trigger)') + plt.legend(loc="upper right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + plt.grid() + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"{trigger_label}_time_integrated_surface_spectra_unnormalized_{station_id}_{run_label}.pdf")) + plt.close() + +def plot_time_integrated_surface_spectra_normalized(station_id, norm_spec_arr, freqs, upward_channels, downward_channels, save_location, run_label, use_monitoring = False, run_event_counts = None): + '''Plot time-integrated normalized surface channel spectra for FORCE trigger events. Use weighted average if use_monitoring is True and run_event_counts is provided, otherwise use simple average.''' + plt.figure(figsize=(10, 6)) + trigger_label = "force" + weights, unit_label = get_weights_if_monitoring(trigger_label, use_monitoring, run_event_counts) + for ch in upward_channels: + if weights is not None: + spec_mean = np.average(norm_spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(norm_spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (up)', linestyle='-') + for ch in downward_channels: + if weights is not None: + spec_mean = np.average(norm_spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(norm_spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (down)', linestyle='--') + + periodiccolor2 = "mediumseagreen" + excesscolor = 'grey' + wb_color = "mediumvioletred" + normcolor = "steelblue" + + plt.axvspan(80, 120, color=excesscolor, alpha=0.3, label="_nolegend_") + plt.axvline(x=0.403e3, color=wb_color, linestyle='--', linewidth=1.2, label="_nolegend_", alpha=0.7) + plt.axvspan(0.278e3, 0.285e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.482e3, 0.485e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.240e3, 0.272e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.360e3, 0.380e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.136e3, 0.139e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.151e3, 0.157e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.125e3, 0.127e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(300, 350, color=normcolor, alpha=0.3, label="_nolegend_") + + plt.xlabel('Frequency [MHz]') + plt.xlim(50, 800) + plt.ylim(0, ) + plt.ylabel(f'Normalized Amplitude Spectrum [a.u.]') + plt.title(f'Time-Integrated Spectrum of Surface Channels (FORCE Trigger)') + + ax = plt.gca() + line_legend = ax.legend( + loc="upper right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + + annotation_handles = [ + Patch(facecolor=excesscolor, alpha=0.3, label="Galactic Excess"), + Line2D([0], [0], color=wb_color, linestyle="--", linewidth=1.2, label="Weather Balloon"), + Patch(facecolor=periodiccolor2, alpha=0.3, label="Periodic Signal"), + Patch(facecolor=normcolor, alpha=0.3, label="Normalization Region"),] + + annotation_legend = ax.legend( + handles=annotation_handles, + loc="lower right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + + ax.add_artist(line_legend) + + plt.grid() + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"time_integrated_surface_spectra_normalized_force_trigger_{station_id}_{run_label}.pdf")) + plt.close() + +def plot_time_integrated_deep_spectra(station_id, spec_arr, freqs, vpol_channels, hpol_channels, save_location, run_label, trigger_label, use_monitoring = False, run_event_counts = None, daq_type="radiant"): + '''Plot time-integrated deep channel spectra. Use weighted average if use_monitoring is True and run_event_counts is provided, otherwise use simple average.''' + plt.figure(figsize=(10, 6)) + weights, unit_label = get_weights_if_monitoring(trigger_label, use_monitoring, run_event_counts, daq_type) + for ch in vpol_channels: + if weights is not None: + spec_mean = np.average(spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (VPOL)', linestyle='-') + for ch in hpol_channels: + if weights is not None: + spec_mean = np.average(spec_arr[ch, :, :], axis=0, weights=weights) + else: + spec_mean = np.mean(spec_arr[ch, :, :], axis=0) + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (HPOL)', linestyle='--') + + plt.xlabel('Frequency [MHz]') + plt.xlim(50, 800) + #plt.ylim(0, 5) + plt.ylabel(f'Amplitude Spectrum [{unit_label}]') + plt.title(f'Time-Integrated Spectrum of Deep Channels ({trigger_label} Trigger)') + plt.legend(loc="upper right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + plt.grid() + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"{trigger_label}_time_integrated_deep_spectra_unnormalized_{station_id}_{run_label}.pdf")) + plt.close() + +def plot_time_integrated_surface_spectra_normalized_example_reference(station_id, norm_spec_arr, freqs, upward_channels, downward_channels, save_location, run_label, use_monitoring = False, run_event_counts = None): + '''Plot time-integrated normalized surface channel spectra for FORCE trigger events. Use weighted average if use_monitoring is True and run_event_counts is provided, otherwise use simple average.''' + up_color = "#0072B2" # blue + down_color = "#D55E00" # vermillion + plt.figure(figsize=(10, 6)) + trigger_label = "force" + weights, unit_label = get_weights_if_monitoring(trigger_label, use_monitoring, run_event_counts) + ch = upward_channels[2] # Just plot the first upward channel as an example + down_channel = downward_channels[1] # Just plot the first downward channel as an example + ref_channels = [12, 14, 19] + if weights is not None: + spec_mean = np.average(norm_spec_arr[ch, :, :], axis=0, weights=weights) #shape (n_freqs,) + spec_mean_down = np.average(norm_spec_arr[down_channel, :, :], axis=0, weights=weights) #shape (n_freqs,) + else: + spec_mean = np.mean(norm_spec_arr[ch, :, :], axis=0) + spec_mean_down = np.mean(norm_spec_arr[down_channel, :, :], axis=0) + ref_mean = np.mean(norm_spec_arr[ref_channels, :, :], axis=(0, 1)) #shape (n_freqs,) + + plt.plot(freqs[1:] / units.MHz, spec_mean[1:], label=f'Ch {ch} (up)', linestyle='-', color=up_color) + plt.plot(freqs[1:] / units.MHz, spec_mean_down[1:], label=f'Ch {down_channel} (down)', linestyle='-.', color=down_color) + plt.plot(freqs[1:] / units.MHz, ref_mean[1:], label=f'Reference Spectrum', linestyle='--', color='black') + + periodiccolor2 = "mediumseagreen" + excesscolor = 'grey' + wb_color = "mediumvioletred" + normcolor = "steelblue" + + plt.axvspan(80, 120, color=excesscolor, alpha=0.3, label="_nolegend_") + plt.axvline(x=0.403e3, color=wb_color, linestyle='--', linewidth=1.2, label="_nolegend_", alpha=0.7) + plt.axvspan(0.278e3, 0.285e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.482e3, 0.485e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.240e3, 0.272e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.360e3, 0.380e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.136e3, 0.139e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.151e3, 0.157e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(0.125e3, 0.127e3, color=periodiccolor2, alpha=0.3, label="_nolegend_") + plt.axvspan(300, 350, color=normcolor, alpha=0.3, label="_nolegend_") + + plt.xlabel('Frequency [MHz]') + plt.xlim(50, 800) + plt.ylim(0, ) + plt.ylabel(f'Normalized Amplitude Spectrum [a.u.]') + #plt.title(f'Time-Integrated Spectrum of Surface Channels (FORCE Trigger)') + + ax = plt.gca() + line_legend = ax.legend( + loc="upper right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + + annotation_handles = [ + Patch(facecolor=excesscolor, alpha=0.3, label="Galactic Excess"), + Line2D([0], [0], color=wb_color, linestyle="--", linewidth=1.2, label="Weather Balloon"), + Patch(facecolor=periodiccolor2, alpha=0.3, label="Periodic Signal"), + Patch(facecolor=normcolor, alpha=0.3, label="Normalization Region"),] + + annotation_legend = ax.legend( + handles=annotation_handles, + loc="lower right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black") + + ax.add_artist(line_legend) + + plt.grid() + plt.tight_layout() + plt.savefig(os.path.join(save_location, f"example_with_reference_spectrum_time_integrated_surface_spectra_normalized_force_trigger_{station_id}_{run_label}.pdf")) + plt.close() \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_trigger_rate.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_trigger_rate.py new file mode 100644 index 0000000..272499b --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_trigger_rate.py @@ -0,0 +1,143 @@ +import matplotlib.pyplot as plt +from matplotlib.lines import Line2D +from matplotlib.patches import Patch +import matplotlib.dates as mdates +import numpy as np +from NuRadioReco.utilities import units +import os +import logging + +logger = logging.getLogger(__name__) + +TRIGGER_LABEL_MAPPING_RADIANT = { + "force_trigger_rate": "FORCE", + "lt_trigger_rate": "LT", + "rf0_trigger_rate": "RADIANT0", + "rf1_trigger_rate": "RADIANT1" +} + +TRIGGER_LABEL_MAPPING_DIDAQ = { + "force_trigger_rate": "FORCE", + "lt_trigger_rate": "DIDAQ_DEEP_PHASED", + "rf0_trigger_rate": "DIDAQ_SURF_UP", + "rf1_trigger_rate": "DIDAQ_SURF_DOWN" +} + +def plot_trigger_rates_over_time(run_trigger_rates, save_location, station_id, run_label, daq_type): + '''Plot trigger rates over time for different trigger types.''' + + fig, ax = plt.subplots(figsize=(10, 6)) + + if daq_type == "didaq": + trigger_label_mapping = TRIGGER_LABEL_MAPPING_DIDAQ + + elif daq_type == "radiant": + trigger_label_mapping = TRIGGER_LABEL_MAPPING_RADIANT + + else: + raise ValueError(f"Unsupported daq_type: {daq_type}") + + trigger_types = [trigger for trigger in trigger_label_mapping.keys()] + trigger_labels = [trigger_label_mapping[trigger] for trigger in trigger_types] + print(f"Trigger types: {trigger_types}, Trigger labels: {trigger_labels}") + + run_numbers = sorted(run_trigger_rates.keys()) + times = [run_trigger_rates[run_no]["run_start_time_utc"] for run_no in run_numbers] + + for trigger_type, trigger_label in zip(trigger_types, trigger_labels): + rates = [run_trigger_rates[run_no].get(trigger_type, np.nan) for run_no in run_numbers] + ax.plot(times, rates, marker="o", label=trigger_label) + + ax.set_xlabel("Run Start Time (UTC)") + ax.set_ylabel("Trigger Rate (Hz)") + ax.set_title(f"Trigger Rates Over Time ({run_label})") + + ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y-%m-%d\n%H:%M")) + fig.autofmt_xdate() + + ax.legend( + loc="upper right", + frameon=True, + fancybox=True, + framealpha=0.9, + edgecolor="black" + ) + + ax.grid() + fig.tight_layout() + + fig.savefig(os.path.join(save_location, f"trigger_rates_over_time_{station_id}_{run_label}.pdf")) + plt.close(fig) + +def plot_trigger_rate_heatmap(run_trigger_rates, save_location, station_id, run_label, daq_type): + '''Plot trigger rates as heatmap (run vs trigger type).''' + + if daq_type == "didaq": + trigger_label_mapping = TRIGGER_LABEL_MAPPING_DIDAQ + + elif daq_type == "radiant": + trigger_label_mapping = TRIGGER_LABEL_MAPPING_RADIANT + + else: + raise ValueError(f"Unsupported daq_type: {daq_type}") + + trigger_types = [trigger for trigger in trigger_label_mapping.keys()] + trigger_labels = [trigger_label_mapping[trigger] for trigger in trigger_types] + + run_numbers = sorted(run_trigger_rates.keys()) + n_runs = len(run_numbers) + + rate_matrix = [] + + for run_no in run_numbers: + rate_matrix.append([ + run_trigger_rates[run_no].get(trigger, np.nan) + for trigger in trigger_types + ]) + + + rate_matrix = np.asarray(rate_matrix) + + times = [ + run_trigger_rates[run_no]["run_start_time_utc"] + for run_no in run_numbers + ] + + fig, ax = plt.subplots(figsize=(10, max(5, len(run_numbers) * 0.08))) + + im = ax.imshow( + rate_matrix, + aspect="auto", + origin="lower" + ) + + cbar = fig.colorbar(im) + cbar.set_label("Trigger Rate (Hz)") + + ax.set_xticks(np.arange(len(trigger_types))) + ax.set_xticklabels(trigger_labels) + + if n_runs <= 15: + ax.set_yticks(np.arange(len(run_numbers))) + ax.set_yticklabels(run_numbers) + else: + ax.set_yticks(np.arange(0, len(run_numbers), max(1, len(run_numbers) // 15))) + ax.set_yticklabels([run_numbers[i] for i in range(0, len(run_numbers), max(1, len(run_numbers) // 15))]) + + + ax.set_xlabel("Trigger Type") + ax.set_ylabel("Run Number") + ax.set_title( + f"Trigger Rate Heatmap ({run_label})" + ) + + fig.tight_layout() + + fig.savefig( + os.path.join( + save_location, + f"trigger_rate_heatmap_{station_id}_{run_label}.pdf" + ) + ) + + plt.close(fig) \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_vrms.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_vrms.py new file mode 100644 index 0000000..74ae76e --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/plotting_functions_sva/plotting_sva_vrms.py @@ -0,0 +1,658 @@ +import os +import matplotlib.pyplot as plt +import numpy as np +import matplotlib.dates as mdates +from datetime import timezone +import logging +import pandas as pd +import matplotlib.dates as mdates +from matplotlib.lines import Line2D + + +logger = logging.getLogger(__name__) + +#### Vrms Plots #### +def choose_day_interval(times): + times = pd.to_datetime(times, utc=True) + total_days = (times.max() - times.min()).days + + if total_days < 10: + return 1 + elif total_days < 20: + return 2 + elif total_days < 40: + return 4 + elif total_days < 80: + return 7 + elif total_days < 150: + return 10 + elif total_days < 300: + return 15 + elif total_days < 600: + return 30 + else: + return 60 + +def plot_vrms_values_against_time(times, vrms_arr_all, channel_list, station_id, run_label, save_location, force_mask, radiant0_mask, radiant1_mask, lt_mask, daq_type, n_rows = 12, n_cols = 2, day_interval=None, use_monitoring=False): + '''Plot RMS (for monitoring.root) or Vrms (for dataProviderRNOG) distributions for different trigger types.''' + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "RMS" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "Vrms" + + if daq_type == "didaq": + trigger_masks = {"FORCE": force_mask, + "DIDAQ_SURF_UP": radiant0_mask, + "DIDAQ_SURF_DOWN": radiant1_mask, + "DIDAQ_DEEP_PHASED": lt_mask,} + trigger_colors = {"FORCE": "tab:blue", + "DIDAQ_SURF_UP": "tab:orange", + "DIDAQ_SURF_DOWN": "tab:green", + "DIDAQ_DEEP_PHASED": "tab:red",} + + elif daq_type == "radiant": + trigger_masks = {"FORCE": force_mask, + "RADIANT0": radiant0_mask, + "RADIANT1": radiant1_mask, + "LT": lt_mask,} + trigger_colors = {"FORCE": "tab:blue", + "RADIANT0": "tab:orange", + "RADIANT1": "tab:green", + "LT": "tab:red",} + + times = np.asarray(times) + vrms_arr_all = np.asarray(vrms_arr_all) + + if day_interval is None: + day_interval = choose_day_interval(times) + + n_channels = len(channel_list) + + fig, axs = plt.subplots(n_rows, n_cols, figsize=(15, 24), sharex=True, squeeze=False) + axs = axs.ravel() + + legend_handles = {} + + for idx, ch in enumerate(channel_list): + ax = axs[idx] + vrms_ch = vrms_arr_all[ch] + + for trig_name, trig_mask in trigger_masks.items(): + times_trig = times[trig_mask] + vrms_trig = vrms_ch[trig_mask] + + scatter = ax.scatter(times_trig, vrms_trig, s=8, alpha=0.5, label=trig_name, color=trigger_colors[trig_name], rasterized=True) + if trig_name not in legend_handles: + legend_handles[trig_name] = scatter + + + ax.set_title(f"Channel {ch}") + ax.grid(alpha=0.4) + + for j in range(len(channel_list), len(axs)): + axs[j].set_visible(False) + + ticks_ax = axs[-1] + time_span = times.max() - times.min() + time_span_days = time_span / np.timedelta64(1, "D") + + if time_span_days < 1: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span_days < 3: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=12)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ticks_ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + fig.legend(handles=[legend_handles[k] for k in trigger_masks.keys()], labels=list(trigger_masks.keys()), + loc="lower center", ncol=4, frameon=True, markerscale=2, bbox_to_anchor=(0.5, 0.005)) + + fig.supylabel(unit_label, x=0.02) + plt.subplots_adjust(bottom = 0.11, wspace = 0.38, left=0.08) + fig.supxlabel("Date [UTC]", x = 0.5, y = 0.06) + plt.savefig(os.path.join(save_location, f"{plot_label.lower()}_against_time_{station_id}_{run_label}.pdf")) + plt.close(fig) + +def plot_vrms_values_against_time_per_trigger(times, vrms_arr_all, channel_list, station_id, run_label, save_location,force_mask, radiant0_mask, radiant1_mask, lt_mask, daq_type, n_rows=12, n_cols=2, day_interval=None, use_monitoring=False): + '''Plot RMS/Vrms against time separately for each trigger type.''' + + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "rms" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "vrms" + + if daq_type == "didaq": + trigger_masks = {"FORCE": force_mask, + "DIDAQ_SURF_UP": radiant0_mask, + "DIDAQ_SURF_DOWN": radiant1_mask, + "DIDAQ_DEEP_PHASED": lt_mask,} + trigger_colors = {"FORCE": "tab:blue", + "DIDAQ_SURF_UP": "tab:orange", + "DIDAQ_SURF_DOWN": "tab:green", + "DIDAQ_DEEP_PHASED": "tab:red",} + + elif daq_type == "radiant": + trigger_masks = {"FORCE": force_mask, + "RADIANT0": radiant0_mask, + "RADIANT1": radiant1_mask, + "LT": lt_mask,} + trigger_colors = {"FORCE": "tab:blue", + "RADIANT0": "tab:orange", + "RADIANT1": "tab:green", + "LT": "tab:red",} + + times = np.asarray(times) + vrms_arr_all = np.asarray(vrms_arr_all) + + if day_interval is None: + day_interval = choose_day_interval(times) + + for trig_name, trig_mask in trigger_masks.items(): + + times_trig = times[trig_mask] + + if len(times_trig) == 0: + logger.warning(f"No {trig_name} events found. Skipping plot.") + continue + + fig, axs = plt.subplots( + n_rows, n_cols, + figsize=(15, 24), + sharex=True, + squeeze=False + ) + axs = axs.ravel() + + for idx, ch in enumerate(channel_list): + ax = axs[idx] + + vrms_trig = vrms_arr_all[ch, trig_mask] + + ax.scatter( + times_trig, + vrms_trig, + s=8, + alpha=0.5, + color=trigger_colors[trig_name], + rasterized=True + ) + + ax.set_title(f"Channel {ch}") + ax.grid(alpha=0.4) + + for j in range(len(channel_list), len(axs)): + axs[j].set_visible(False) + + ticks_ax = axs[len(channel_list) - 1] + + time_span = times_trig.max() - times_trig.min() + time_span_days = time_span / np.timedelta64(1, "D") + + if time_span_days < 1: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + elif time_span_days < 3: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + else: + ticks_ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d", tz=timezone.utc) + ) + + fig.suptitle(f"{unit_label} vs Time — {trig_name}", y=0.995) + fig.supylabel(unit_label, x=0.02) + + plt.subplots_adjust(bottom = 0.11, wspace = 0.38, left=0.08) + fig.supxlabel("Date [UTC]", x = 0.5, y = 0.06) + + filename = f"{plot_label}_against_time_{trig_name.lower()}_{station_id}_{run_label}.pdf" + plt.savefig(os.path.join(save_location, filename)) + plt.close(fig) + +def plot_vrms_values_against_time_single_trigger_zscore(times, vrms_arr, flag, z_score, k_values, trigger_name, channel_list, station_id, run_label, save_location, + n_rows=12, n_cols=2, day_interval=None, use_monitoring=False): + '''Plot RMS/Vrms against time for a single trigger type with z-score outlier highlighting.''' + + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "rms" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "vrms" + + times = pd.to_datetime(times, utc=True) + if day_interval is None: + day_interval = choose_day_interval(times) + + fig, axs = plt.subplots( + n_rows, + n_cols, + figsize=(15, 24), + sharex=True, + squeeze=False + ) + + axs = axs.ravel() + + for idx, ch in enumerate(channel_list): + + ax = axs[idx] + + vrms_ch = vrms_arr[ch] + flag_ch = flag[ch] + + good_mask = ~flag_ch + + ax.scatter( + times[good_mask], + vrms_ch[good_mask], + s=8, + alpha=0.25, + color="gray", + rasterized=True + ) + + zex = np.abs(z_score[ch]) - k_values[ch] + zex = np.clip(zex, 0, None) + + sc = ax.scatter( + times[flag_ch], + vrms_ch[flag_ch], + s=8, + c=zex[flag_ch], + cmap="Reds", + rasterized=True + ) + + if np.any(flag_ch): + cax = ax.inset_axes([1.02, 0.1, 0.05, 0.8]) + plt.colorbar(sc, cax=cax, label=r"$|z|-k$") + + ax.grid(alpha=0.4) + + ax.text( + 0.85, + 0.95, + f"Ch {ch}", + transform=ax.transAxes, + ha="left", + va="top", + bbox=dict( + boxstyle="round, pad=0.25", + facecolor="white", + alpha=0.8 + ) + ) + + for j in range(len(channel_list), len(axs)): + axs[j].set_visible(False) + + red = plt.cm.Reds(0.6) + + legend_handles = [ + Line2D( + [0], [0], + marker="o", + color="none", + markeredgecolor="gray", + markerfacecolor="gray", + alpha=0.4, + markersize=6, + label=r"$|z|\leq k$" + ), + Line2D( + [0], [0], + marker="o", + color="none", + markeredgecolor=red, + markerfacecolor=red, + markersize=6, + label=r"$|z|>k$" + ) + ] + + axs[0].legend(handles=legend_handles, loc="upper left") + + ticks_ax = axs[len(channel_list) - 1] + + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + if time_span < 1: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + + elif time_span < 3: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + + else: + ticks_ax.xaxis.set_major_locator( + mdates.DayLocator(interval=day_interval) + ) + + ticks_ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d", tz=timezone.utc) + ) + + fig.autofmt_xdate() + + fig.suptitle(f"{unit_label} vs Time — {trigger_name}", y=0.995) + + fig.supylabel(unit_label, x=0.02) + + plt.subplots_adjust(bottom = 0.11, wspace = 0.38, left=0.08) + fig.supxlabel("Date [UTC]", x = 0.5, y = 0.06) + + filename = ( + f"{plot_label}_against_time_" + f"{trigger_name.lower()}_" + f"{station_id}_{run_label}.pdf" + ) + + plt.savefig(os.path.join(save_location, filename)) + + plt.close(fig) + +def plot_vrms_values_against_time_single_channel_zscore( + times, + vrms_arr, + flag, + z_score, + k_values, + trigger_name, + channel, + station_id, + run_label, + save_location, + day_interval=None, + use_monitoring=False +): + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "rms" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "vrms" + + times = pd.to_datetime(times, utc=True) + + if day_interval is None: + day_interval = choose_day_interval(times) + + fig, ax = plt.subplots(figsize=(10,5)) + + vrms_ch = vrms_arr[channel] + flag_ch = flag[channel] + + good_mask = ~flag_ch + + ax.scatter( + times[good_mask], + vrms_ch[good_mask], + s=8, + alpha=0.25, + color="gray", + rasterized=True + ) + + zex = np.abs(z_score[channel]) - k_values[channel] + zex = np.clip(zex,0,None) + + sc = ax.scatter( + times[flag_ch], + vrms_ch[flag_ch], + s=8, + c=zex[flag_ch], + cmap="Reds", + rasterized=True + ) + + if np.any(flag_ch): + plt.colorbar(sc, ax=ax, label=r"$|z|-k$") + + red = plt.cm.Reds(0.6) + + legend_handles = [ + Line2D( + [0],[0], + marker="o", + color="none", + markeredgecolor="gray", + markerfacecolor="gray", + alpha=0.4, + markersize=6, + label=r"$|z|\leq k$" + ), + Line2D( + [0],[0], + marker="o", + color="none", + markeredgecolor=red, + markerfacecolor=red, + markersize=6, + label=r"$|z|>k$" + ) + ] + + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + if time_span < 1: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + elif time_span < 3: + ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc) + ) + else: + ax.xaxis.set_major_locator( + mdates.DayLocator(interval=day_interval) + ) + ax.xaxis.set_major_formatter( + mdates.DateFormatter("%m-%d", tz=timezone.utc) + ) + + ax.set_xlabel("Date [UTC]") + ax.set_ylabel(unit_label) + ax.grid(alpha=0.4) + + ax.text( + 0.85, + 0.95, + f"Ch {channel}", + transform=ax.transAxes, + ha="left", + va="top", + bbox=dict( + boxstyle="round, pad=0.25", + facecolor="white", + alpha=0.8 + ) + ) + + ax.legend(handles=legend_handles, loc="upper left") + #ax.set_title(f"{unit_label} vs Time — {trigger_name}") + + fig.autofmt_xdate() + plt.tight_layout() + + filename = ( + f"{plot_label}_against_time_" + f"{trigger_name.lower()}_" + f"ch{channel}_" + f"{station_id}_{run_label}.pdf" + ) + + plt.savefig(os.path.join(save_location,filename)) + + plt.close(fig) + +def plot_rolling_mean_std(times, rolling_mean_arr, rolling_std_arr, channel_list, station_id, run_label, trigger_name, save_location, n_rows=12, n_cols=2, day_interval=None, use_monitoring=False): + '''Plot rolling mean and std for Vrms values against time for each channel.''' + times = pd.to_datetime(times, utc=True) + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "rms" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "vrms" + + fig, axs = plt.subplots(n_rows, n_cols, figsize=(15, 24), sharex=True, squeeze=False) + axs = axs.ravel() + + for idx, ch in enumerate(channel_list): + ax = axs[idx] + + ax.plot(times, rolling_mean_arr[ch], label="Rolling Mean", color="blue") + ax.plot(times, rolling_std_arr[ch], label="Rolling Std", color="orange") + + ax.set_title(f"Channel {ch}") + ax.grid(alpha=0.4) + ax.legend() + + for j in range(len(channel_list), len(axs)): + axs[j].set_visible(False) + + ticks_ax = axs[len(channel_list) - 1] + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + if day_interval is None: + day_interval = choose_day_interval(times) + + if time_span < 1: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span < 3: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ticks_ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + fig.autofmt_xdate() + fig.supylabel(f"Rolling Value ({unit_label})", x=0.02) + + plt.subplots_adjust(bottom = 0.11, wspace = 0.38, left=0.08) + fig.supxlabel("Date [UTC]", x = 0.5, y = 0.06) + + plt.savefig(os.path.join(save_location, f"{plot_label}_rolling_mean_std_against_time_{trigger_name}_{station_id}_{run_label}.pdf")) + plt.close(fig) + +def plot_rolling_mean_linregress(times, rolling_mean_arr, channel_list, slope_dict, intercept_dict, station_id, run_label, trigger_name, save_location, n_rows=12, n_cols=2, day_interval=None, use_monitoring=False): + '''Plot rolling mean for Vrms values against time for each channel with linear regression line.''' + times = pd.to_datetime(times, utc=True) + + if day_interval is None: + day_interval = choose_day_interval(times) + + if use_monitoring: + unit_label = "RMS [ADC]" + plot_label = "rms" + else: + unit_label = r"$V_\mathrm{rms}$ [V]" + plot_label = "vrms" + + times_rel_hour = (times - times.min()).total_seconds() / 3600.0 + + fig, axs = plt.subplots(n_rows, n_cols, figsize=(15, 24), sharex=True, squeeze=False) + axs = axs.ravel() + + for idx, ch in enumerate(channel_list): + ax = axs[idx] + + ax.plot(times, rolling_mean_arr[ch], label="Rolling Mean", color="blue") + + slope = slope_dict[ch] + intercept = intercept_dict[ch] + reg_line = slope * times_rel_hour + intercept + ax.plot(times, reg_line, label="Linear Fit", color="red", linestyle="--") + + ax.set_title(f"Channel {ch}") + ax.grid(alpha=0.4) + ax.legend() + + for j in range(len(channel_list), len(axs)): + axs[j].set_visible(False) + + ticks_ax = axs[len(channel_list) - 1] + time_span = (times.max() - times.min()).total_seconds() / 86400.0 + + if time_span < 1: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=3)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + elif time_span < 3: + ticks_ax.xaxis.set_major_locator(mdates.HourLocator(interval=6)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d\n%H:%M", tz=timezone.utc)) + else: + ticks_ax.xaxis.set_major_locator(mdates.DayLocator(interval=day_interval)) + ticks_ax.xaxis.set_major_formatter(mdates.DateFormatter("%m-%d", tz=timezone.utc)) + + fig.autofmt_xdate() + fig.supylabel(f"Rolling Mean ({unit_label})", x=0.02) + + plt.subplots_adjust(bottom = 0.11, wspace = 0.38, left=0.08) + fig.supxlabel("Date [UTC]", x = 0.5, y = 0.06) + + plt.savefig(os.path.join(save_location, f"{plot_label}_rolling_mean_linregress_against_time_{trigger_name}_{station_id}_{run_label}.pdf")) + plt.close(fig) + +def create_heatmap_plot(results_dict, label, save_dir, channel_list, station_id, + matrix_key, run_label, cmap="Reds", vmin=None, vmax=None): + '''Create and save heatmap plots for each channel.''' + + for ch in channel_list: + + matrix = np.array(results_dict[ch][matrix_key]) + runs = list(range(matrix.shape[0])) + + title = f"{label} Heatmap for Channel {ch}" + save_path = os.path.join( + save_dir, + f"force_trigger_station_{station_id}_{label.lower().replace(' ', '_')}_heatmap_channel{ch}_{run_label}.pdf" + ) + + fig, ax = plt.subplots(figsize=(10, 8)) + + im = ax.imshow( + matrix, + cmap=cmap, + vmin=vmin, + vmax=vmax, + origin="upper", + aspect="equal", + rasterized=True + ) + + cbar = plt.colorbar(im, ax=ax) + cbar.set_label(f"{label}") + + ax.set_xticks(np.arange(len(runs))) + ax.set_yticks(np.arange(len(runs))) + + ax.set_xticklabels([]) + ax.set_yticklabels([]) + + ax.set_xlabel("Run Number") + ax.set_ylabel("Run Number") + ax.set_title(title) + + plt.tight_layout() + plt.savefig(save_path, dpi=300) + plt.close() \ No newline at end of file diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/science_verification_analysis_main.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/science_verification_analysis_main.py new file mode 100644 index 0000000..6f28a45 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/science_verification_analysis_main.py @@ -0,0 +1,636 @@ +''' +This module can be used to test if the stations are working as expected. See README.md for more details. +''' + +import rnog_data.runtable as rt +import logging +import os +import datetime +import numpy as np +from matplotlib import pyplot as plt +from argparse import ArgumentParser +import pandas as pd +import matplotlib.dates as mdates +from datetime import timezone +import copy +import csv +import json +import random +import string + +# Import config files +from config_files_sva.config_plotting import set_plot_style + +# Import analysis functions +from monitoring_data_functions_sva.get_monitoring_data_uproot import choose_trigger_type_header, read_multiple_runs +from analysis_functions_sva.spectral_analysis_sva import normalize_channels, normalize_channels_new, find_amplitude_ratio_in_band, find_amplitude_ratio_in_band_specific_bkg, excess_info_from_ratio, excess_info_from_ratio_specific_bkg, validate_excess_in_bands +from analysis_functions_sva.z_score_analysis_sva import calculate_statistics_log_paramater, calculate_z_score_parameter, symmetry_metrics_channel_z_score, symmetry_metrics_z_score, load_values_json, outlier_flag, find_outlier_details, calculate_expected_values_per_trigger, outlier_details +from analysis_functions_sva.vrms_analysis_sva import calculate_vrms, kde_modality, tail_fraction_and_trimmed_skew_two_sided, report_vrms_characteristics, get_rms_per_trigger_monitoring +from analysis_functions_sva.glitching_analysis_sva import binomtest_glitch_fraction +from analysis_functions_sva.block_offsets_analysis_sva_monitoring import get_force_block_offsets_monitoring, block_offset_statistics_monitoring, plot_block_offsets_violin_monitoring +from analysis_functions_sva.vrms_stability_analysis_sva import get_rms_per_run, relative_median_shift, decision_metric + +# Import helper functions +from helper_functions.output_writer import write_failed_runs_to_csv, write_spectral_results, write_snr_outlier_details, write_vrms_outlier_details, write_vrms_modality_results, write_glitching_results, write_block_offset_results, block_offset_channel_health, create_result_csv_file, create_result_csv_file_didaq, write_readme_for_shifters +from helper_functions.read_rnog_runtable import read_rnog_runtable +from helper_functions.config_helper import get_station_config + +# Import plotting functions +from plotting_functions_sva.plotting_sva_spectrum import plot_time_integrated_surface_spectra_unnormalized, plot_time_integrated_surface_spectra_normalized, plot_time_integrated_deep_spectra, plot_time_integrated_surface_spectra_normalized_example_reference +from plotting_functions_sva.plotting_sva_snr import plot_snr_against_time_single_channel, choose_day_interval, plot_snr_against_time, plot_snr_against_time_per_trigger +from plotting_functions_sva.plotting_sva_vrms import plot_vrms_values_against_time_single_channel_zscore, plot_vrms_values_against_time, plot_vrms_values_against_time_single_trigger_zscore, create_heatmap_plot, plot_vrms_values_against_time_per_trigger +from plotting_functions_sva.plotting_sva_glitch import glitching_violin_plot, choose_bin_size, plot_glitch_q99_over_time +from plotting_functions_sva.plotting_sva_debug import debug_plot_vrms_distribution_single_channel, debug_plot_ratios, debug_plot_snr_distribution, debug_plot_z_score_snr, debug_plot_vrms_distribution, debug_plot_ratios_just_galaxy +from plotting_functions_sva.plotting_sva_trigger_rate import plot_trigger_rates_over_time, plot_trigger_rate_heatmap + +#### Script directory +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) + +#### Reference and config directories +REFERENCE_DIR = os.path.join(SCRIPT_DIR, "expected_values") +CONFIG_DIR = os.path.join(SCRIPT_DIR, "config_files_sva") + +#### Logging +logger = logging.getLogger(__name__) + +def setup_logging(station_id, run_label, LOGS_DIR): + + log_file = os.path.join(LOGS_DIR, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +def failed_check_plot(validation_arr, label, *plot_fns, note=""): + """Log pass/fail for a channel-health test and run plot_fns(save_dir), routing to + failed_test_plots on failure or detailed_plots otherwise.""" + failed = validation_arr.isin(["X", "!!"]).any() + save_dir = failed_test_plots if failed else detailed_plots + status = "Some channels failed" if failed else "All channels passed" + (logger.warning if failed else logger.info)(f"{status} the {label}.{note} Detailed plots saved in {save_dir}.") + for plot_fn in plot_fns: + plot_fn(save_dir) + +def failed_check_report(validation_arr, label, *report_fns, note=""): + """Log pass/fail for a channel-health test and run report_fns(save_dir), routing to + failed_test_results on failure or detailed_results otherwise.""" + failed = validation_arr.isin(["X", "!!"]).any() + save_dir = failed_test_results_dir if failed else detailed_results_dir + status = "Some channels failed" if failed else "All channels passed" + (logger.warning if failed else logger.info)(f"{status} the {label}.{note} Detailed results saved in {save_dir}.") + for report_fn in report_fns: + report_fn(save_dir) + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis - extracting data from monitoring.root files") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("--data_location", type=str, default="desy", help="Location of the data. Use 'desy' (inbox data), 'uchicago' (mirrored data) or provide a custom path to the data directory, e.g. --data_location /path/to/data") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + args = argparser.parse_args() + + use_monitoring = True + rms_label = "rms" + + station_id = args.station_id + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + # Date and random string for unique save directory + date_label = datetime.datetime.now().strftime("%y-%m-%d") + random_string = ''.join(random.choices(string.digits, k=6)) + + save_directory_label = (f"{date_label}_station-{station_id}_run{first_run}-run{last_run}_{random_string}") + + # Choose the data location based on the argument provided and define the save location for the results + if args.data_location == "desy": + logger.info("Using DESY inbox data location for the analysis.") + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + result_base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/NuRadioMC/science_verification_analysis" + + elif args.data_location == "uchicago": + logger.info("Using UChicago mirrored data location for the analysis.") + base_data_path = "/data/satellite" + result_base_data_path = "/data/sva" + + else: + logger.info(f"Using custom data location {args.data_location} for the analysis.") + base_data_path = args.data_location + #result_base_data_path = os.path.join(args.data_location, "results") + result_base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/NuRadioMC/science_verification_analysis" + + + + result_save_location = os.path.join(result_base_data_path, save_directory_label) + os.makedirs(result_save_location, exist_ok=True) + + logger.info(f"Results will be saved in {result_save_location}.") + + # Output directories for plots, results, and logs + save_location = os.path.join(result_save_location, "plots") + results_dir = os.path.join(result_save_location, "test_results") + csv_dir = os.path.join(result_save_location, "channel_health_summary") + logs_dir = os.path.join(result_save_location, "logs") + + # Create output directories if they don't exist + os.makedirs(results_dir, exist_ok=True) + os.makedirs(save_location, exist_ok=True) + os.makedirs(csv_dir, exist_ok=True) + os.makedirs(logs_dir, exist_ok=True) + + # Path to README for shifters + shifters_readme_file = os.path.join(result_save_location, "README_shifters.txt") + + ## Results directories + # Always save results, even if all channels pass the tests + detailed_results_dir = os.path.join(results_dir, "detailed_results") + os.makedirs(detailed_results_dir, exist_ok=True) + + # Save only if some channels fail the tests + failed_test_results_dir = os.path.join(results_dir, "failed_test_results") + os.makedirs(failed_test_results_dir, exist_ok=True) + + ## Plot directories + # Plot always + standard_plots = os.path.join(save_location, "standard_plots") + os.makedirs(standard_plots, exist_ok=True) + + # Plot only if some channels fail the tests + failed_test_plots = os.path.join(save_location, "failed_test_plots") + os.makedirs(failed_test_plots, exist_ok=True) + + # Debug plots, e.g. heatmaps + debug_plots = os.path.join(save_location, "debug_plots") + os.makedirs(debug_plots, exist_ok=True) + + # Detailed plots, e.g. per channel plots (failed plots will be saved here if no channels fail the tests) + detailed_plots = os.path.join(save_location, "detailed_plots") + os.makedirs(detailed_plots, exist_ok=True) + + # Start logging + setup_logging(station_id, run_label, logs_dir) + logger.info(f"Starting analysis for station {station_id}, runs: {run_numbers} using the monitoring.root files.") + + # Set the plotting style + set_plot_style() + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + surface_channels = config["surface_channels"] + deep_channels = config["deep_channels"] + upward_channels = config["upward_channels"] + downward_channels = config["downward_channels"] + vpol_channels = config["vpol_channels"] + hpol_channels = config["hpol_channels"] + phased_array_channels = config["phased_array_channels"] + all_channels = config["all_channels"] + reference_channels_galaxy = config["reference_channels_galaxy"] + reference_channels = config["reference_channels"] + + # Choose RADIANT or DIDAQ based on the station configuration + digitizer_type = config["daq_type"] + if digitizer_type not in ["radiant", "didaq"]: + logger.error(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'.") + raise ValueError(f"Invalid daq_type {digitizer_type}. Must be either 'radiant' or 'didaq'. Please check the station configuration in config_station.json.") + + if digitizer_type == "radiant": + logger.info(f"Using RADIANT digitizer type for station {station_id}. Triggers are FORCE, LT, RADIANT0, RADIANT1.") + trigger_types_daq = {"force" : "FORCE", "lt" : "LT", "radiant0" : "RADIANT0", "radiant1" : "RADIANT1"} + elif digitizer_type == "didaq": + logger.info(f"Using DIDAQ digitizer type for station {station_id}. Triggers are FORCE, DIDAQ_DEEP_PHASED, DIDAQ_SURF_UP, DIDAQ_SURF_DOWN.") + trigger_types_daq = {"force" : "FORCE", "lt" : "DIDAQ_DEEP_PHASED", "radiant0" : "DIDAQ_SURF_UP", "radiant1" : "DIDAQ_SURF_DOWN"} + + # Load event information from the combined_event_info dictionary: + combined_event_info = read_multiple_runs(base_path = base_data_path, station_id = station_id, run_numbers=run_numbers, daq_type=digitizer_type) + + times = combined_event_info["trigger_time_utc"] + valid_times_mask = ~pd.isna(times) + times = times[valid_times_mask] + invalid_runs = np.unique(combined_event_info["run_no"][~valid_times_mask]) + + run_no = combined_event_info["run_no"][valid_times_mask] + trigger_type_arr = combined_event_info["triggerType"][valid_times_mask] + + max_abs_amplitude_arr = combined_event_info["max_abs_amplitude_arr"][:, valid_times_mask] + event_number_arr = combined_event_info["event_number_arr"][valid_times_mask] + run_event_counts = combined_event_info["run_event_counts"] # dict with run number as key and value as another dict with n_events, n_forced_triggers, n_lt_triggers, n_rf0_triggers, n_rf1_triggers for that run + run_trigger_rates = combined_event_info["run_trigger_rates"] # dict with run number as key and value as another dict with trigger rates for different trigger types for that run + + failed_run_info = combined_event_info["failed_run_info"] or {} # dict with run number as key and value as reason for failure, only for runs that failed to be read + failed_runs = list(failed_run_info.keys()) + + if len(invalid_runs) > 0: + logger.warning(f"Some events have invalid trigger times and will be excluded from the analysis, events belong to the runs: {list(map(int, invalid_runs))}. ") + for invalid_run in invalid_runs: + failed_run_info[int(invalid_run)] = "Some events have been skipped in the analysis due to invalid timestamps, check logs for details" + + + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=results_dir) # Save in the results directory, since always important + + n_events_force = combined_event_info["total_n_force_triggers"] + n_lt_events = combined_event_info["total_n_lt_triggers"] # DIDAQ_DEEP_PHASED for DIDAQ, LT for RADIANT + n_radiant0_events = combined_event_info["total_n_rf0_triggers"] # DIDAQ_SURF_UP for DIDAQ, RADIANT0 for RADIANT + n_radiant1_events = combined_event_info["total_n_rf1_triggers"] # DIDAQ_SURF_DOWN for DIDAQ, RADIANT1 for RADIANT + + # Spectral info: + freqs = combined_event_info["freqs"] + avg_spectrum = combined_event_info["avg_spectrum"] + spec_arr_force = combined_event_info["avg_spectrum_force"] + spec_arr_lt = combined_event_info["avg_spectrum_lt"] # DIDAQ_DEEP_PHASED for DIDAQ, LT for RADIANT + spec_arr_radiant0 = combined_event_info["avg_spectrum_rf0"] # DIDAQ_SURF_UP for DIDAQ, RADIANT0 for RADIANT + spec_arr_radiant1 = combined_event_info["avg_spectrum_rf1"] # DIDAQ_SURF_DOWN for DIDAQ, RADIANT1 for RADIANT + + # Glitching, SNR and block offset info: + rms_arr = combined_event_info["rms_arr"][:, valid_times_mask] + snr_arr = combined_event_info["snr_arr"][:, valid_times_mask] + + if digitizer_type == "radiant": + glitch_arr = combined_event_info["glitching_test_statistic_arr"][:, valid_times_mask] + block_offsets_arr = combined_event_info["block_offsets_arr"][:, valid_times_mask] + + # Choose the day interval for plotting based on the time range of the events + day_interval = choose_day_interval(times) + + # Spectral analysis configuration parameters + spectral_analysis_config_json = os.path.join(CONFIG_DIR, "config_spectral_analysis.json") + with open(spectral_analysis_config_json, "r") as f: + spectral_analysis_config_dict = json.load(f) + + spectral_bands = spectral_analysis_config_dict["spectral_bands"] + alpha_spec = spectral_analysis_config_dict["alpha_spec"] + ci_threshold_spec = spectral_analysis_config_dict["ci_threshold_spec"] + normalization_band = spectral_analysis_config_dict["normalization_band"] + log_ratio_thresholds_spec = spectral_analysis_config_dict["log_ratio_thresholds_spec"] + + # Normalize the spectra for the FORCE trigger events + norm_spec_arr_force, scale_factors_force = normalize_channels_new(spec_arr_force, freqs, downward_channels, upward_channels, normalization_band=normalization_band) + + # Masks for different trigger types + force_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["force"], digitizer_type) # FORCE for both DIDAQ and RADIANT + lt_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["lt"], digitizer_type) # DIDAQ_DEEP_PHASED for DIDAQ, LT for RADIANT + radiant0_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["radiant0"], digitizer_type) # DIDAQ_SURF_UP for DIDAQ, RADIANT0 for RADIANT + radiant1_mask = choose_trigger_type_header(trigger_type_arr, trigger_types_daq["radiant1"], digitizer_type) # DIDAQ_SURF_DOWN for DIDAQ, RADIANT1 for RADIANT + + run_no_force = run_no[force_mask] + event_number_force = event_number_arr[force_mask] + + snr_arr_radiant0 = snr_arr[:, radiant0_mask] + snr_arr_radiant1 = snr_arr[:, radiant1_mask] + snr_arr_lt = snr_arr[:, lt_mask] + + times_radiant0 = times[radiant0_mask] + times_radiant1 = times[radiant1_mask] + times_lt = times[lt_mask] + + # Bands for spectral analysis + band_config = copy.deepcopy(spectral_bands) + + for band_name in band_config: + if band_name != "galactic_excess": + band_config[band_name]["reference_channels"] = reference_channels + elif band_name == "galactic_excess": + band_config[band_name]["reference_channels"] = reference_channels_galaxy + else: + logger.error(f"Unknown band name {band_name} in SPECTRAL_BANDS config") + raise ValueError(f"Unknown band name {band_name} in SPECTRAL_BANDS config") + + logger.debug(f"Band configuration for spectral analysis: {band_config}") + + ratio_arr_dict = find_amplitude_ratio_in_band_specific_bkg(freqs, norm_spec_arr_force, upward_channels, downward_channels, **band_config) + + channels_order = upward_channels + downward_channels + ch_to_idx = {ch: i for i, ch in enumerate(channels_order)} + logger.debug(f"Channel to index mapping: {ch_to_idx}") + + all_excess_info = {} + all_validation_results = {} + + logger.info("Starting spectral analysis for FORCE trigger events. !!! Different methods for monitoring and dataProviderRNOG !!! ") + for ch in surface_channels: + i = ch_to_idx[ch] + ratio_arr_dict_ch = {} + + for band_name, ratio_arr in ratio_arr_dict.items(): + ratio_arr_dict_ch[band_name] = ratio_arr[i] + + excess_info_results = excess_info_from_ratio_specific_bkg(ratio_arr_dict_ch, alpha_spec, ci_threshold_spec, use_monitoring=use_monitoring, log_ratio_thresholds=log_ratio_thresholds_spec) + validation_results = validate_excess_in_bands(excess_info_results) + + all_excess_info[ch] = excess_info_results + all_validation_results[ch] = validation_results + + ###### SNR analysis + logger.info("Starting SNR analysis for FORCE trigger events...") + snr_arr_force = snr_arr[:, force_mask] + times = np.array(times) + times_force = times[force_mask] + + log_snr_arr, log_mean_list, log_median_list, log_std_list, log_difference_list = calculate_statistics_log_paramater(snr_arr_force) + reference_filename = f"expected_snr/expected_snr_values_station{station_id}.json" + k_values_log_snr, ref_log_mean_list, ref_log_std_list = load_values_json(REFERENCE_DIR, reference_filename) + z_score_arr_log_snr = calculate_z_score_parameter(log_snr_arr, ref_log_mean_list, ref_log_std_list, all_channels) + flag_outliers_snr = outlier_flag(z_score_arr_log_snr, k_values_log_snr, all_channels) + + outlier_details_snr = find_outlier_details(z_score_arr_log_snr, k_values_log_snr, flag_outliers_snr, all_channels, run_no_force, event_number_force) + + ##### Vrms analysis + logger.info("Starting Vrms analysis for monitoring data...") + + rms_arr, rms_arr_force, rms_arr_radiant0, rms_arr_radiant1, rms_arr_lt = get_rms_per_trigger_monitoring(rms_arr=rms_arr, force_mask=force_mask, lt_mask=lt_mask, radiant0_mask=radiant0_mask, radiant1_mask=radiant1_mask) + + logger.info(f"Number of {trigger_types_daq['radiant0']} trigger events: {len(rms_arr_radiant0[1])}, Number of {trigger_types_daq['radiant1']} trigger events: {len(rms_arr_radiant1[1])}, Number of {trigger_types_daq['lt']} trigger events: {len(rms_arr_lt[1])}") + logger.info(f"Calculating RMS modality and tail characteristics for each trigger type...") + + # Load the configuration parameters for the RMS analysis from the JSON file + rms_config_json = os.path.join(CONFIG_DIR, "config_rms.json") + with open(rms_config_json, "r") as f: + rms_config_dict = json.load(f) + + kde_modality_function_parameters = rms_config_dict["kde_modality_function_parameters"] + skewness_function_parameters = rms_config_dict["skewness_function_parameters"] + report_vrms_function_parameters = rms_config_dict["report_vrms_function_parameters"] + + modality_dict_force = kde_modality(rms_arr_force, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_force = tail_fraction_and_trimmed_skew_two_sided(rms_arr_force, all_channels, skewness_config=skewness_function_parameters) + if len(rms_arr_force[1]) < 100: + logger.warning(f"{trigger_types_daq['force']} trigger has less than 100 valid RMS entries ({len(rms_arr_force[1])}). Results for the Vrms statistics may be unreliable.") + modality_force, tail_label_force = report_vrms_characteristics(modality_dict_force, tail_dict_force, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_radiant0 = kde_modality(rms_arr_radiant0, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_radiant0 = tail_fraction_and_trimmed_skew_two_sided(rms_arr_radiant0, all_channels, skewness_config=skewness_function_parameters) + if len(rms_arr_radiant0[1]) < 100: + logger.warning(f"{trigger_types_daq['radiant0']} trigger has less than 100 valid RMS entries ({len(rms_arr_radiant0[1])}). Results for the Vrms statistics may be unreliable.") + modality_radiant0, tail_label_radiant0 = report_vrms_characteristics(modality_dict_radiant0, tail_dict_radiant0, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_radiant1 = kde_modality(rms_arr_radiant1, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_radiant1 = tail_fraction_and_trimmed_skew_two_sided(rms_arr_radiant1, all_channels, skewness_config=skewness_function_parameters) + if len(rms_arr_radiant1[1]) < 100: + logger.warning(f"{trigger_types_daq['radiant1']} trigger has less than 100 valid RMS entries ({len(rms_arr_radiant1[1])}). Results for the Vrms statistics may be unreliable.") + modality_radiant1, tail_label_radiant1 = report_vrms_characteristics(modality_dict_radiant1, tail_dict_radiant1, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_lt = kde_modality(rms_arr_lt, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_lt = tail_fraction_and_trimmed_skew_two_sided(rms_arr_lt, all_channels, skewness_config=skewness_function_parameters) + if len(rms_arr_lt[1]) < 100: + logger.warning(f"{trigger_types_daq['lt']} trigger has less than 100 valid RMS entries ({len(rms_arr_lt[1])}). Results for the Vrms statistics may be unreliable.") + + modality_lt, tail_label_lt = report_vrms_characteristics(modality_dict_lt, tail_dict_lt, all_channels, report_config=report_vrms_function_parameters) + + ## Vrms stability + reference_filename_rms = f"expected_{rms_label}/expected_{rms_label}_station{station_id}.json" + vrms_k_values, vrms_ref_mean, vrms_ref_std = load_values_json(REFERENCE_DIR, reference_filename_rms) + z_score_arr_vrms_force = calculate_z_score_parameter(rms_arr_force, vrms_ref_mean, vrms_ref_std, all_channels) + flag_outliers_vrms_force = outlier_flag(z_score_arr_vrms_force, vrms_k_values, all_channels) + outlier_details_vrms_force = find_outlier_details(z_score_arr_vrms_force, vrms_k_values, flag_outliers_vrms_force, channel_list=all_channels, run_no=run_no_force, event_number=event_number_force) + + rms_arr_per_run_dict_force = get_rms_per_run(rms_arr_force, run_no_force) + relative_median_shift_results = relative_median_shift(rms_arr_per_run_dict_force, all_channels) + + rms_results = decision_metric(outlier_details_vrms_force, relative_median_shift_results, n_events_force=n_events_force, channels=all_channels) + + ##### Glitching and block offset analysis - only for RADIANT digitizer type + if digitizer_type == "radiant": + ##### Glitching analysis + logger.info("Starting glitching analysis...") + + # Load the configuration parameters for the glitching analysis from the JSON file + glitching_config_json = os.path.join(CONFIG_DIR, "config_glitching.json") + with open(glitching_config_json, "r") as f: + glitching_config_dict = json.load(f) + + config_glitching = glitching_config_dict["config_glitching_values"] + + glitch_info = binomtest_glitch_fraction(glitch_arr, all_channels, config_glitching=config_glitching) + + ##### Block offsets analysis + # Get the reference block offset results for the station + ref_block_offset_results_file = os.path.join(REFERENCE_DIR, "expected_block_offsets", f"expected_block_offsets_station{station_id}.json") + with open(ref_block_offset_results_file, "r") as f: + ref_block_offset_results = json.load(f) + + logger.info("Starting block offset analysis (monitoring.root), results are not used to determine channel health, see warnings in the log file for channels with potential block offset issues. The block offsets are then removed.") + block_offset_arr_force = get_force_block_offsets_monitoring(block_offsets_arr, force_mask) + block_offset_stats = block_offset_statistics_monitoring(block_offset_arr_force=block_offset_arr_force, channel_list=all_channels) + + block_offset_results_dict = block_offset_channel_health(block_offset_stats, ref_block_off_dict=ref_block_offset_results, use_monitoring=use_monitoring) + + + # Create summary CSV file + if digitizer_type == "radiant": + results_df = create_result_csv_file( + station_id, + run_label, + n_events_force, + surface_channels, + downward_channels, + upward_channels, + all_channels, + all_validation_results, + glitch_info, + block_offset_results_dict, + rms_results, + modality_dict_force, + modality_dict_lt, + modality_dict_radiant0, + modality_dict_radiant1, + outlier_details_snr, + csv_dir, + rms_label + ) + rms_modality_lt_validation_arr = results_df[f"{rms_label.capitalize()} (LT)"] + rms_modality_radiant0_validation_arr = results_df[f"{rms_label.capitalize()} (RADIANT0)"] + rms_modality_radiant1_validation_arr = results_df[f"{rms_label.capitalize()} (RADIANT1)"] + glitching_validation_arr = results_df["Glitching"] + block_offset_validation_arr = results_df["Block Offsets"] + + elif digitizer_type == "didaq": + results_df = create_result_csv_file_didaq( + station_id, + run_label, + n_events_force, + surface_channels, + downward_channels, + upward_channels, + all_channels, + all_validation_results, + rms_results, + modality_dict_force, + modality_dict_lt, + modality_dict_radiant0, + modality_dict_radiant1, + outlier_details_snr, + csv_dir, + rms_label + ) + rms_modality_lt_validation_arr = results_df[f"{rms_label.capitalize()} (DEEP PHASED)"] + rms_modality_radiant0_validation_arr = results_df[f"{rms_label.capitalize()} (SURF UP)"] + rms_modality_radiant1_validation_arr = results_df[f"{rms_label.capitalize()} (SURF DOWN)"] + + snr_validation_arr = results_df["SNR"] + galaxy_validation_arr = results_df["Galaxy (FORCE)"] + rms_stability_validation_arr = results_df[f"{rms_label.capitalize()} Stability (FORCE)"] + rms_modality_force_validation_arr = results_df[f"{rms_label.capitalize()} (FORCE)"] + + + #### Plotting #### + + #### Standard plots for the analysis results + # FORCE trigger spectra - normalized, unnormalized + plot_time_integrated_surface_spectra_normalized(station_id, norm_spec_arr_force, freqs, upward_channels, downward_channels, standard_plots, run_label, use_monitoring=use_monitoring, run_event_counts=run_event_counts) + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_force, freqs, upward_channels, downward_channels, standard_plots, run_label, trigger_label="force", use_monitoring=use_monitoring, run_event_counts=run_event_counts, daq_type=digitizer_type) + plot_time_integrated_deep_spectra(station_id, spec_arr_force, freqs, vpol_channels, hpol_channels, standard_plots, run_label, trigger_label="force", use_monitoring=use_monitoring, run_event_counts=run_event_counts, daq_type=digitizer_type) + + # SNR against time (FORCE trigger) + plot_snr_against_time(station_id, times_force, snr_arr_force, flag_outliers_snr, z_score_arr_log_snr, k_values_log_snr, all_channels, standard_plots, run_label, nrows=12, ncols=2, day_interval=day_interval) + + # FORCE trigger RMS against time + plot_vrms_values_against_time_single_trigger_zscore(times_force, rms_arr_force, flag_outliers_vrms_force, z_score_arr_vrms_force, vrms_k_values, trigger_name="FORCE", channel_list=all_channels, station_id=station_id, run_label=run_label, save_location=standard_plots, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + + # Trigger rate plots over time for all trigger types + plot_trigger_rates_over_time(run_trigger_rates, standard_plots, station_id, run_label, daq_type = digitizer_type) + + #### Debug plots/reports for the analysis results, routed by pass/fail status of each test + # SNR test: plot SNR distribution and z-score distribution, write outlier details to text file + failed_check_plot(snr_validation_arr, "SNR test", + lambda d: debug_plot_snr_distribution(log_snr_arr, channel_list=all_channels, save_location=d, station_id=station_id, run_label=run_label, bins=30), + lambda d: debug_plot_z_score_snr(z_score_arr_log_snr, channel_list=all_channels, save_location=d, station_id=station_id, run_label=run_label, bins=30)) + + failed_check_report(snr_validation_arr, "SNR test", + lambda d: write_snr_outlier_details(outlier_details_snr, station_id, run_label, n_events_force, results_dir=d)) + + # Galaxy test: plot ratios for each channel, write spectral results to text file + failed_check_plot(galaxy_validation_arr, "Galaxy test for FORCE trigger", + lambda d: debug_plot_ratios(ratio_arr_dict=ratio_arr_dict, channels_order=channels_order, save_location=d, station_id=station_id, run_label=run_label, bins=30)) + + failed_check_report(galaxy_validation_arr, "Galaxy test for FORCE trigger", + lambda d: write_spectral_results(all_excess_info, surface_channels, station_id, run_label, results_dir=d)) + + # RMS stability test: plot relative median shift heatmap, write results to text file + failed_check_report(rms_stability_validation_arr, "RMS stability test", + lambda d: write_vrms_outlier_details(outlier_details_vrms_force, station_id, run_label, trigger_label="FORCE", n_events=n_events_force, results_dir=d, use_monitoring=use_monitoring), + lambda d: (lambda f: json.dump(relative_median_shift_results, f, indent=4))(open(os.path.join(d, f"{rms_label}_relative_median_shift_results_force_trigger_station{station_id}_{run_label}.json"), "w")), + lambda d: (lambda f: json.dump(rms_results, f, indent=4))(open(os.path.join(d, f"rms_stability_decision_results_force_trigger_station{station_id}_{run_label}.json"), "w"))) + + + # RMS modality tests: FORCE is used for overall channel health, the others are informational only + rms_modality_checks = [ + (rms_modality_force_validation_arr, "FORCE", rms_arr_force, modality_dict_force, ""), + (rms_modality_lt_validation_arr, trigger_types_daq["lt"], rms_arr_lt, modality_dict_lt, " (isn't included in overall channel health but might indicate a problem)"), + (rms_modality_radiant0_validation_arr, trigger_types_daq["radiant0"], rms_arr_radiant0, modality_dict_radiant0, " (isn't included in overall channel health but might indicate a problem)"), + (rms_modality_radiant1_validation_arr, trigger_types_daq["radiant1"], rms_arr_radiant1, modality_dict_radiant1, " (isn't included in overall channel health but might indicate a problem)"), + ] + for validation_arr, trigger_label, rms_arr_trig, modality_dict, note in rms_modality_checks: + failed_check_plot(validation_arr, f"{rms_label} modality test for {trigger_label} trigger", + lambda d, rms_arr_trig=rms_arr_trig, modality_dict=modality_dict, trigger_label=trigger_label: debug_plot_vrms_distribution(rms_arr_trig, modality_dict, channel_list=all_channels, station_id=station_id, run_label=run_label, trigger_label=trigger_label, save_location=d, n_rows=12, n_cols=2, use_monitoring=use_monitoring), + note=note) + + failed_check_report(rms_modality_force_validation_arr, f"{rms_label} modality test for FORCE trigger", + lambda d: write_vrms_modality_results(modality_force, tail_label_force, trigger_label=trigger_types_daq['force'], station_id=station_id, run_label=run_label, results_dir=d, use_monitoring=use_monitoring), + lambda d: write_vrms_modality_results(modality_radiant0, tail_label_radiant0, trigger_label=trigger_types_daq['radiant0'], station_id=station_id, run_label=run_label, results_dir=d, use_monitoring=use_monitoring), + lambda d: write_vrms_modality_results(modality_radiant1, tail_label_radiant1, trigger_label=trigger_types_daq['radiant1'], station_id=station_id, run_label=run_label, results_dir=d, use_monitoring=use_monitoring), + lambda d: write_vrms_modality_results(modality_lt, tail_label_lt, trigger_label=trigger_types_daq['lt'], station_id=station_id, run_label=run_label, results_dir=d, use_monitoring=use_monitoring)) + + + if digitizer_type == "radiant": + failed_check_plot(glitching_validation_arr, "glitching test", + lambda d: plot_glitch_q99_over_time(np.array(times), glitch_arr, all_channels, station_id, run_label, d), + lambda d: glitching_violin_plot(glitch_arr, all_channels, station_id, run_label, d)) + + failed_check_report(glitching_validation_arr, "glitching test", + lambda d: write_glitching_results(glitch_info, station_id, run_label, all_channels, results_dir=d)) + + failed_check_plot(block_offset_validation_arr, "block offset test", + lambda d: plot_block_offsets_violin_monitoring(block_offset_arr_force, all_channels, station_id, run_label, d)) + + failed_check_report(block_offset_validation_arr, "block offset test", + lambda d: write_block_offset_results(block_offset_stats, station_id, run_label, ref_block_off_dict=ref_block_offset_results, results_dir=d, use_monitoring=use_monitoring)) + + #### Other plots (always saved to other_debug_plots) + for trig_key in ("lt", "radiant0", "radiant1"): + spec_arr_trig = {"lt": spec_arr_lt, "radiant0": spec_arr_radiant0, "radiant1": spec_arr_radiant1}[trig_key] + trigger_label = trigger_types_daq[trig_key] + # Surface spectrum + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_trig, freqs, upward_channels, downward_channels, detailed_plots, run_label, trigger_label=trigger_label, use_monitoring=use_monitoring, run_event_counts=run_event_counts, daq_type=digitizer_type) + # Deep spectrum (unnormalized) + plot_time_integrated_deep_spectra(station_id, spec_arr_trig, freqs, vpol_channels, hpol_channels, detailed_plots, run_label, trigger_label=trigger_label, use_monitoring=use_monitoring, run_event_counts=run_event_counts, daq_type=digitizer_type) + + # RMS + plot_vrms_values_against_time(times, rms_arr, all_channels, station_id, run_label, detailed_plots, force_mask, radiant0_mask, radiant1_mask, lt_mask, daq_type=digitizer_type, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + plot_vrms_values_against_time_per_trigger(times, rms_arr, all_channels, station_id, run_label, detailed_plots, force_mask, radiant0_mask, radiant1_mask, lt_mask, daq_type=digitizer_type, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + + # Trigger rate plots + plot_trigger_rate_heatmap(run_trigger_rates, detailed_plots, station_id, run_label, daq_type=digitizer_type) + + # SNR + for times_trig, snr_arr_trig, color, trig_key in ( + (times_radiant0, snr_arr_radiant0, "tab:orange", "radiant0"), + (times_radiant1, snr_arr_radiant1, "tab:green", "radiant1"), + (times_lt, snr_arr_lt, "tab:red", "lt"), + ): + plot_snr_against_time_per_trigger(station_id, times_trig, snr_arr_trig, all_channels, detailed_plots, run_label, nrows=12, ncols=2, day_interval=day_interval, color=color, triggerlabel=trigger_types_daq[trig_key]) + + # Debug plots - for now only median RMS shift heatmaps + create_heatmap_plot(relative_median_shift_results, label="Relative Median Shift", save_dir=debug_plots, channel_list=all_channels, station_id=station_id, matrix_key="median_shift_matrix", run_label=run_label, cmap="Reds") + + logger.info(f"Analysis completed for station {station_id}, run label {run_label}. Results saved in {results_dir}. Standard plots saved in {standard_plots}. Detailed plots saved in {detailed_plots}. Plots for failed tests saved in {failed_test_plots}. Debug plots saved in {debug_plots}. Summary CSV saved in {csv_dir}. Logs saved in {logs_dir}.") + + ## Write README for shifters + + write_readme_for_shifters(shifters_readme_file, station_id, run_numbers, times, run_label) + + + diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/read_rnog_data_nuradio.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/read_rnog_data_nuradio.py new file mode 100644 index 0000000..0ed5b25 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/read_rnog_data_nuradio.py @@ -0,0 +1,175 @@ +import os +import numpy as np +from tqdm import tqdm +import NuRadioReco.framework.event +import NuRadioReco.framework.station +import NuRadioReco.framework.channel +import NuRadioReco.framework.trigger +from NuRadioReco.framework.parameters import channelParameters as chp +from NuRadioReco.modules.RNO_G.dataProviderRNOG import dataProviderRNOG +from NuRadioReco.framework.parameters import channelParametersRNOG as chp_rnog +from NuRadioReco.utilities import units +from collections import defaultdict +import logging + +logger = logging.getLogger(__name__) + +def convert_events_information(event_info, convert_to_arrays=True): + + data = defaultdict(list) + + for ele in event_info.values(): + for k, v in ele.items(): + data[k].append(v) + + if convert_to_arrays: + for k in data: + data[k] = np.array(data[k]) + + return data + +def read_rnog_data(station_id: int, run_numbers: list, backend: str = "pyroot", sampling_rate = 2.4*units.GHz): + '''Read RNO-G data for a given station and list of run numbers using the specified backend.''' + file_list = [] + valid_run_numbers = [] + missing_runs = [] + + for run_id in run_numbers: + path = f"/pnfs/ifh.de/acs/radio/diskonly/data/inbox/station{station_id}/run{run_id}/combined.root" + if os.path.isfile(path): + file_list.append(path) + valid_run_numbers.append(run_id) + else: + missing_runs.append(run_id) + + if missing_runs: + logger.warning(f"!!!! Skipping {len(missing_runs)} missing runs: {missing_runs} !!!!") + if not file_list: + raise FileNotFoundError("No combined.root files found for selected runs.") + + n_files = len(file_list) + n_batches = n_files // 100 + 1 + logger.info(f"Reading {n_files} files in {n_batches} batches using {backend} backend.") + + event_info = defaultdict(list) + + n_events_total = 0 + spec_batches = [] + trace_batches = [] + times_trace_batches = [] + snr_batches = [] + glitch_batches = [] + block_offset_batches = [] + run_no_all = [] + times_all = [] + freqs = None + + from NuRadioReco.modules.channelSignalReconstructor import channelSignalReconstructor as csr + csr = csr() + csr.begin(debug=False) + + from NuRadioReco.modules.RNO_G.channelGlitchDetector import channelGlitchDetector as cgd_rnog + cgd_rnog = cgd_rnog() + cgd_rnog.begin() + + from NuRadioReco.modules.RNO_G.channelBlockOffsetFitter import channelBlockOffsets as cbo + cbo = cbo() + + for batch in tqdm(np.array_split(np.array(file_list), n_batches), desc="Reading batches", unit="batch"): + tableReader = dataProviderRNOG() + tableReader.begin(files=batch.tolist(), + det=None, + reader_kwargs={"overwrite_sampling_rate":sampling_rate, + "convert_to_voltage":True, + "apply_baseline_correction":"auto", + "mattak_kwargs":{"backend":backend}}) + event_info_tmp = tableReader.reader.get_events_information( + keys=["triggerType", "triggerTime", "readoutTime", "radiantThrs", "lowTrigThrs", "run", "eventNumber"]) + + event_info_tmp = convert_events_information(event_info_tmp, False) + for key, value in event_info_tmp.items(): + event_info[key] += value + + n_events = tableReader.reader.get_n_events() + n_events_total += n_events + logger.info(f"Reading {n_events} events in this batch.") + + channel_list = [i for i in range(24)] + spec_arr = np.zeros((len(channel_list), n_events, 1025)) + trace_arr = np.zeros((len(channel_list), n_events, 2048)) + times_trace_arr = np.zeros((len(channel_list), n_events, 2048)) + snr_arr = np.zeros((len(channel_list), n_events)) + glitch_arr = np.zeros((len(channel_list), n_events)) + block_offset_arr = np.zeros((len(channel_list), n_events, 16)) + + run_no = [] + times = [] + event_ids = [] + + for idx, event in enumerate(tqdm(tableReader.run(), total=n_events, desc="Events", unit="evt", leave=False)): + station = event.get_station() + time = station.get_station_time().datetime64 + times.append(time) + run_no.append(event.get_run_number()) + + csr.run(evt=event, station=station, det=None, stored_noise=False) + cgd_rnog.run(event=event, station=station, det=None) + for i_ch, ch in enumerate(channel_list): + channel = station.get_channel(ch) + + times_ch = channel.get_times() + times_trace_arr[i_ch, idx, :] = times_ch + + snr_dict = channel.get_parameter(chp.SNR) + snr_peak = snr_dict["peak_amplitude"] + snr_arr[i_ch, idx] = snr_peak + + glitching_values = channel.get_parameter(chp_rnog.glitch_test_statistic) + glitch_arr[i_ch, idx] = glitching_values + + spec = channel.get_frequency_spectrum() + spec_arr[i_ch, idx, :] = np.abs(spec) + + trace = channel.get_trace() + trace_arr[i_ch, idx, :] = trace + + block_offsets = channel.get_parameter(chp.block_offsets) + block_offset_arr[i_ch, idx, :] = block_offsets + + if freqs is None and idx == 0 and i_ch == 0: + freqs = channel.get_frequencies() + + spec_batches.append(spec_arr) + trace_batches.append(trace_arr) + times_trace_batches.append(times_trace_arr) + snr_batches.append(snr_arr) + glitch_batches.append(glitch_arr) + block_offset_batches.append(block_offset_arr) + run_no_all.extend(run_no) + times_all.extend(times) + + #tableReader.end() + + spec_arr = np.concatenate(spec_batches, axis=1) + trace_arr = np.concatenate(trace_batches, axis=1) + times_trace_arr = np.concatenate(times_trace_batches, axis=1) + snr_arr = np.concatenate(snr_batches, axis=1) + glitch_arr = np.concatenate(glitch_batches, axis=1) + block_offset_arr = np.concatenate(block_offset_batches, axis=1) + + run_no = np.array(run_no_all) + times = np.array(times_all) + + for key, value in event_info.items(): + event_info[key] = np.array(value) + + inf_mask = np.isinf(event_info["triggerTime"]) + event_info["triggerTime"][inf_mask] = event_info["readoutTime"][inf_mask] + if np.any(inf_mask): + logger.warning(f"Found {np.sum(inf_mask)} events with inf trigger time (of {len(inf_mask)} events)") + + logger.info(f"n_events read: {spec_arr.shape[1]}, n_events_total: {n_events_total}") + logger.debug(f"freqs shape: {freqs.shape}, spec_arr shape: {spec_arr.shape}, trace_arr shape: {trace_arr.shape}, times_trace_arr shape: {times_trace_arr.shape}, snr_arr shape: {snr_arr.shape}, times shape: {times.shape}, run_no shape: {run_no.shape}, block_offset_arr shape: {block_offset_arr.shape}") + logger.debug(f"trigger types: {np.unique(event_info['triggerType'])}") + + return spec_arr, trace_arr, times_trace_arr, snr_arr, run_no, times, freqs, event_info, glitch_arr, block_offset_arr diff --git a/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/science_verification_analysis_dataprovider.py b/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/science_verification_analysis_dataprovider.py new file mode 100644 index 0000000..2ce0539 --- /dev/null +++ b/rnog_analysis_tools/data_monitoring/science_verification_analysis/sva_dataproviderrnog/science_verification_analysis_dataprovider.py @@ -0,0 +1,463 @@ +''' +This module is an outdated version of the science verification analysis. Instead of using the monitoring data, it uses dataProviderRNOG() to extract information from full waveforms. +It only reads events from the combined.root files, which contain only a subset of the full dataset. Data reading is much slower than the monitoring data. +It is kept here for archival purposes and can be used if there are no monitoring.root files available for the chosen dataset. The analysis won't be updated anymore and might contain different/outdated methods than the current version. +!!!! Reference value files does not exist for this method, so the analysis will not be able to calculate z-scores and outlier flags for SNR and Vrms. Please first calculate expected values using the scripts under /analysis-tools/rnog_analysis_tools/data_monitoring/science_verification_analysis/expected_values/outdated and change the file paths!!!! +!!!! RMS analysis is done in ADC units, which is not correct for this analysis. The results will be wrong and should not be used. Please use the monitoring.root files for the RMS analysis. !!!! +''' + +import rnog_data.runtable as rt +import logging +import os +import datetime +import numpy as np +from matplotlib import pyplot as plt +from argparse import ArgumentParser +import pandas as pd +import matplotlib.dates as mdates +from datetime import timezone +import copy +import csv +import json +from NuRadioReco.utilities import units +import sys + +#### Script directory +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +PARENT_DIR = os.path.dirname(SCRIPT_DIR) +sys.path.insert(0, PARENT_DIR) + +#### Output directories for plots, results, and logs +PLOTS_DIR = os.path.join(SCRIPT_DIR, "plots") +RESULTS_DIR = os.path.join(SCRIPT_DIR, "detailed_results") +CSV_DIR = os.path.join(SCRIPT_DIR, "channel_health_summary") +LOGS_DIR = os.path.join(SCRIPT_DIR, "logs") + +REFERENCE_DIR = os.path.join(PARENT_DIR, "expected_values", "outdated") +CONFIG_DIR = os.path.join(PARENT_DIR, "config_files_sva") + +# Create output directories if they don't exist +os.makedirs(RESULTS_DIR, exist_ok=True) +os.makedirs(PLOTS_DIR, exist_ok=True) +os.makedirs(CSV_DIR, exist_ok=True) +os.makedirs(LOGS_DIR, exist_ok=True) + +# Import config files +from config_files_sva.config_plotting import set_plot_style + +# Import analysis functions +from read_rnog_data_nuradio import convert_events_information, read_rnog_data +from monitoring_data_functions_sva.get_monitoring_data_uproot import choose_trigger_type_header, read_multiple_runs +from analysis_functions_sva.spectral_analysis_sva import normalize_channels, find_amplitude_ratio_in_band, find_amplitude_ratio_in_band_specific_bkg, excess_info_from_ratio, excess_info_from_ratio_specific_bkg, validate_excess_in_bands +from analysis_functions_sva.z_score_analysis_sva import calculate_statistics_log_paramater, calculate_z_score_parameter, symmetry_metrics_channel_z_score, symmetry_metrics_z_score, load_values_json, outlier_flag, find_outlier_details, calculate_expected_values_per_trigger, outlier_details +from analysis_functions_sva.vrms_analysis_sva import calculate_vrms, kde_modality, tail_fraction_and_trimmed_skew_two_sided, report_vrms_characteristics, get_rms_per_trigger_monitoring +from analysis_functions_sva.glitching_analysis_sva import binomtest_glitch_fraction +from analysis_functions_sva.block_offsets_analysis_sva_dataproviderrnog import get_block_offsets_after_removal, get_block_offsets_before_removal, plot_block_offsets_violin_before_after_comparison, block_offset_statistics +from analysis_functions_sva.block_offsets_analysis_sva_monitoring import get_force_block_offsets_monitoring, block_offset_statistics_monitoring, plot_block_offsets_violin_monitoring +from analysis_functions_sva.vrms_stability_analysis_sva import get_rms_per_run, relative_median_shift, decision_metric + +# Import helper functions +from helper_functions.output_writer import write_failed_runs_to_csv, write_spectral_results, write_snr_outlier_details, write_vrms_outlier_details, write_vrms_modality_results, write_glitching_results, write_block_offset_results, create_result_csv_file +from helper_functions.read_rnog_runtable import read_rnog_runtable +from helper_functions.config_helper import get_station_config + +# Import plotting functions +from plotting_functions_sva.plotting_sva_spectrum import plot_time_integrated_surface_spectra_unnormalized, plot_time_integrated_surface_spectra_normalized, plot_time_integrated_deep_spectra, plot_time_integrated_surface_spectra_normalized_example_reference +from plotting_functions_sva.plotting_sva_snr import choose_day_interval, plot_snr_against_time, plot_snr_against_time_per_trigger +from plotting_functions_sva.plotting_sva_vrms import plot_vrms_values_against_time, plot_vrms_values_against_time_single_trigger_zscore, create_heatmap_plot, plot_vrms_values_against_time_per_trigger +from plotting_functions_sva.plotting_sva_glitch import glitching_violin_plot, choose_bin_size, plot_glitch_q99_over_time +from plotting_functions_sva.plotting_sva_debug import debug_plot_ratios, debug_plot_snr_distribution, debug_plot_z_score_snr, debug_plot_vrms_distribution +from plotting_functions_sva.plotting_sva_trigger_rate import plot_trigger_rates_over_time, plot_trigger_rate_heatmap + + +#### Logging +logger = logging.getLogger(__name__) + +def setup_logging(station_id, run_label): + + log_file = os.path.join(LOGS_DIR, f"logging_science_verification_analysis_station{station_id}_{run_label}.log") + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + handlers=[ + logging.FileHandler(log_file, mode="w"), + logging.StreamHandler() + ], + force=True + ) + + logger.info(f"Logging to {log_file}") + +#### Choose events based on trigger type for dataProviderRNOG case +def choose_trigger_type(event_info, trigger_type: str): + '''Choose events based on trigger type.''' + mask = event_info["triggerType"] == trigger_type + + return mask + + +if __name__ == "__main__": + + argparser = ArgumentParser(description="RNO-G Science Verification Analysis using dataProviderRNOG() !!!! Outdated, please use science_verification_analysis_main.py !!!!") + + argparser.add_argument("-st", "--station_id", type=int, required=True, help="Station to analyze, e.g --station_id 14") + argparser.add_argument("-b", "--backend", type=str, default="pyroot", help="!!! Only needed for method 'monitoring' !!!. Backend to use for reading data, should be either pyroot or uproot (default: pyroot), e.g. --backend pyroot or --backend uproot") + argparser.add_argument("-sl", "--save_location", type=str, default=PLOTS_DIR, help="Location to save the output plots (default: plots directory under script directory), e.g. --save_location /path/to/save/plots") + argparser.add_argument("-ex", "--exclude-runs", nargs="+", type=int, default=[], metavar="RUN", help="Run number(s) to exclude, e.g. --exclude-runs 1005 1010") + argparser.add_argument("--debug_plot", action="store_true", help="If set, will create debug plots.") + argparser.add_argument("--sampling_rate", type=str, default= "after_2024", choices=["before_2024", "after_2024"], help="!!! Only needed for method 'monitoring' !!!. Sampling rate to use, choices are 'before_2024' (3.2 GHz) and 'after_2024' (2.4 GHz), default is 'after_2024'.") + + run_selection = argparser.add_mutually_exclusive_group(required=True) + run_selection.add_argument("--runs", nargs="+", type=int, metavar="RUN_NUMBERS", + help="Run number(s) to analyze. Each run number should be given explicitly separated by a space, e.g. --runs 1001 1002 1005") + run_selection.add_argument("--run_range", nargs=2, type=int, metavar=("START_RUN", "END_RUN"), + help="Range of run numbers to analyze (inclusive). Provide start and end run numbers separated by a space, e.g. --run_range 1000 1050") + run_selection.add_argument("--time_range", nargs=2, type=str, metavar=("START_DATE", "END_DATE"), + help="Date range to analyze (inclusive). Provide start and end dates separated by a space in YYYY-MM-DD format, e.g. --time_range 2024-07-15 2024-09-30") + + + args = argparser.parse_args() + + use_monitoring = False + + logger.info("Using dataProviderRNOG method to read data !!!! Outdated, please use science_verification_analysis_main.py !!!!") + rms_label = "vrms" + + station_id = args.station_id + backend = args.backend + if backend not in ["pyroot", "uproot"]: + raise ValueError("Backend should be either 'pyroot' or 'uproot'") + + if args.runs: + run_numbers = args.runs + elif args.run_range: + run_numbers = list(range(args.run_range[0], args.run_range[1] + 1)) + elif args.time_range: + start_time, stop_time = args.time_range + runtable = read_rnog_runtable(station_id, start_time, stop_time) + run_numbers = runtable["run"].tolist() + else: + raise ValueError("No run selection provided") + + # Exclude specified runs + if args.exclude_runs: + exclude_set = set(args.exclude_runs) + run_numbers = [r for r in run_numbers if r not in exclude_set] + + run_numbers = sorted(run_numbers) + first_run = run_numbers[0] + last_run = run_numbers[-1] + + if first_run == last_run: + run_label = f"run_{first_run}" + else: + run_label = f"runs_{first_run}_{last_run}" + + sampling_rate_choice = args.sampling_rate + sampling_rate = {"after_2024": 2.4*units.GHz, + "before_2024": 3.2*units.GHz} + sr = sampling_rate[sampling_rate_choice] + + # Start logging + setup_logging(station_id, run_label) + logger.info(f"Starting analysis for station {station_id}, runs: {run_numbers}, backend: {backend}, sampling rate: {sr}") + # Set the plotting style + set_plot_style() + + # Create save location directory if it doesn't exist + save_location = os.path.expanduser(args.save_location) + os.makedirs(save_location, exist_ok=True) + + # Get channel lists from config + station_config_json = os.path.join(CONFIG_DIR, "config_station.json") + with open(station_config_json, "r") as f: + station_config_data = json.load(f) + + default_station_config = station_config_data.get("default_config", {}) + station_specific_adjustments = station_config_data.get("station_specific_adjustments", {}) + config = get_station_config(station_id, default_station_config, station_specific_adjustments) + + surface_channels = config["surface_channels"] + deep_channels = config["deep_channels"] + upward_channels = config["upward_channels"] + downward_channels = config["downward_channels"] + vpol_channels = config["vpol_channels"] + hpol_channels = config["hpol_channels"] + phased_array_channels = config["phased_array_channels"] + all_channels = config["all_channels"] + reference_channels_galaxy = config["reference_channels_galaxy"] + reference_channels = config["reference_channels"] + + base_data_path = "/pnfs/ifh.de/acs/radio/diskonly/data/inbox/" + + # Read data + spec_arr, trace_arr, times_trace_arr, snr_arr, run_no, times, freqs, event_info, glitch_arr, block_offsets_arr = read_rnog_data(station_id, run_numbers, backend=backend, sampling_rate=sr) + + # Spectral analysis configuration parameters + spectral_analysis_config_json = os.path.join(CONFIG_DIR, "config_spectral_analysis.json") + with open(spectral_analysis_config_json, "r") as f: + spectral_analysis_config_dict = json.load(f) + + spectral_bands = spectral_analysis_config_dict["spectral_bands"] + alpha_spec = spectral_analysis_config_dict["alpha_spec"] + ci_threshold_spec = spectral_analysis_config_dict["ci_threshold_spec"] + normalization_band = spectral_analysis_config_dict["normalization_band"] + log_ratio_thresholds_spec = spectral_analysis_config_dict["log_ratio_thresholds_spec"] + + # Normalize surface channel spectra + norm_spec_arr, scale_factors = normalize_channels(spec_arr, freqs, downward_channels, upward_channels, normalization_band=normalization_band) + logger.debug(f"Event info trigger type: {event_info['triggerType']}") + logger.debug(f"Spec arr shape: {spec_arr.shape}, Norm spec arr shape: {norm_spec_arr.shape}") + + # Select FORCE trigger events + force_mask = choose_trigger_type(event_info, "FORCE") + + ###### Spectral analysis for FORCE trigger events only + spec_arr_force = spec_arr[:, force_mask, :] + norm_spec_arr_force = norm_spec_arr[:, force_mask, :] + logger.info(f"Number of FORCE trigger events: {spec_arr_force.shape[1]}") + + if len(spec_arr_force[1]) < 30: + logger.warning("Less than 30 FORCE-trigger events, results of the sign test may not be reliable.") + lt_mask = choose_trigger_type(event_info, "LT") + spec_arr_lt = spec_arr[:, lt_mask, :] + + radiant0_mask = choose_trigger_type(event_info, "RADIANT0") + spec_arr_radiant0 = spec_arr[:, radiant0_mask, :] + + radiant1_mask = choose_trigger_type(event_info, "RADIANT1") + spec_arr_radiant1 = spec_arr[:, radiant1_mask, :] + run_event_counts = None # Not available when reading with dataProviderRNOG, only with monitoring data, used for spectral plotting + run_no_force = event_info["run"][force_mask] + event_number_force = event_info["eventNumber"][force_mask] + n_events_force = spec_arr_force.shape[1] + + failed_run_info = {} + excluded_runs = args.exclude_runs.copy() if args.exclude_runs else [] + if excluded_runs: + logger.info(f"Excluding runs {excluded_runs} from the analysis as specified by the user.") + for excluded_run in excluded_runs: + failed_run_info[int(excluded_run)] = "Run excluded by user" + + if failed_run_info: + write_failed_runs_to_csv(station_id, failed_run_info, run_label, results_dir=RESULTS_DIR) + + # Bands for spectral analysis + band_config = copy.deepcopy(spectral_bands) + + for band_name in band_config: + if band_name != "galactic_excess": + band_config[band_name]["reference_channels"] = reference_channels + elif band_name == "galactic_excess": + band_config[band_name]["reference_channels"] = reference_channels_galaxy + else: + logger.error(f"Unknown band name {band_name} in SPECTRAL_BANDS config") + raise ValueError(f"Unknown band name {band_name} in SPECTRAL_BANDS config") + + logger.debug(f"Band configuration for spectral analysis: {band_config}") + + ratio_arr_dict = find_amplitude_ratio_in_band_specific_bkg(freqs, norm_spec_arr_force, upward_channels, downward_channels, **band_config) + + channels_order = upward_channels + downward_channels + ch_to_idx = {ch: i for i, ch in enumerate(channels_order)} + logger.debug(f"Channel to index mapping: {ch_to_idx}") + + all_excess_info = {} + all_validation_results = {} + + logger.info("Starting spectral analysis for FORCE trigger events. !!! Different methods for monitoring and dataProviderRNOG !!! ") + for ch in surface_channels: + i = ch_to_idx[ch] + ratio_arr_dict_ch = {} + + for band_name, ratio_arr in ratio_arr_dict.items(): + ratio_arr_dict_ch[band_name] = ratio_arr[i] + + excess_info_results = excess_info_from_ratio_specific_bkg(ratio_arr_dict_ch, alpha_spec, ci_threshold_spec, use_monitoring=use_monitoring, log_ratio_thresholds=log_ratio_thresholds_spec) + validation_results = validate_excess_in_bands(excess_info_results) + + all_excess_info[ch] = excess_info_results + all_validation_results[ch] = validation_results + + # Write detailed spectral results for all channels in a single write (dCache/pnfs is write-once) + write_spectral_results(all_excess_info, surface_channels, station_id, run_label, results_dir=RESULTS_DIR) + + # Surface spectrum + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_force, freqs, upward_channels, downward_channels, save_location, run_label, trigger_label="force", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_lt, freqs, upward_channels, downward_channels, save_location, run_label, trigger_label="lt", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_radiant0, freqs, upward_channels, downward_channels, save_location, run_label, trigger_label="radiant0", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_surface_spectra_unnormalized(station_id, spec_arr_radiant1, freqs, upward_channels, downward_channels, save_location, run_label, trigger_label="radiant1", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + + # Normalized surface spectrum - only force + plot_time_integrated_surface_spectra_normalized(station_id, norm_spec_arr_force, freqs, upward_channels, downward_channels, save_location, run_label, use_monitoring = use_monitoring, run_event_counts = run_event_counts) + + # Deep spectrum (unnormalized) + plot_time_integrated_deep_spectra(station_id, spec_arr_force, freqs, vpol_channels, hpol_channels, save_location, run_label, trigger_label="force", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_deep_spectra(station_id, spec_arr_lt, freqs, vpol_channels, hpol_channels, save_location, run_label, trigger_label="lt", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_deep_spectra(station_id, spec_arr_radiant0, freqs, vpol_channels, hpol_channels, save_location, run_label, trigger_label="radiant0", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + plot_time_integrated_deep_spectra(station_id, spec_arr_radiant1, freqs, vpol_channels, hpol_channels, save_location, run_label, trigger_label="radiant1", use_monitoring = use_monitoring, run_event_counts = run_event_counts) + + ###### SNR analysis + logger.info("Starting SNR analysis for FORCE trigger events...") + snr_arr_force = snr_arr[:, force_mask] + times = np.array(times) + times_force = times[force_mask] + + log_snr_arr, log_mean_list, log_median_list, log_std_list, log_difference_list = calculate_statistics_log_paramater(snr_arr_force) + reference_filename = f"expected_snr/expected_snr_values_station{station_id}.json" + k_values_log_snr, ref_log_mean_list, ref_log_std_list = load_values_json(REFERENCE_DIR, reference_filename) + z_score_arr_log_snr = calculate_z_score_parameter(log_snr_arr, ref_log_mean_list, ref_log_std_list, all_channels) + flag_outliers_snr = outlier_flag(z_score_arr_log_snr, k_values_log_snr, all_channels) + + outlier_details_snr = find_outlier_details(z_score_arr_log_snr, k_values_log_snr, flag_outliers_snr, all_channels, run_no_force, event_number_force) + write_snr_outlier_details(outlier_details_snr, station_id, run_label, n_events_force, results_dir=RESULTS_DIR) + + day_interval = choose_day_interval(times) + plot_snr_against_time(station_id, times_force, snr_arr_force, flag_outliers_snr, z_score_arr_log_snr, k_values_log_snr, all_channels, save_location, run_label, nrows=12, ncols=2, day_interval=day_interval) + + snr_arr_radiant0 = snr_arr[:, radiant0_mask] + snr_arr_radiant1 = snr_arr[:, radiant1_mask] + snr_arr_lt = snr_arr[:, lt_mask] + + times_radiant0 = times[radiant0_mask] + times_radiant1 = times[radiant1_mask] + times_lt = times[lt_mask] + + plot_snr_against_time_per_trigger(station_id, times_radiant0, snr_arr_radiant0, all_channels, save_location, run_label, nrows=12, ncols=2, day_interval=day_interval, color = "tab:orange", triggerlabel="RADIANT0") + plot_snr_against_time_per_trigger(station_id, times_radiant1, snr_arr_radiant1, all_channels, save_location, run_label, nrows=12, ncols=2, day_interval=day_interval, color = "tab:green", triggerlabel="RADIANT1") + plot_snr_against_time_per_trigger(station_id, times_lt, snr_arr_lt, all_channels, save_location, run_label, nrows=12, ncols=2, day_interval=day_interval, color = "tab:red", triggerlabel="LT") + + ##### Vrms analysis + rms_config_json = os.path.join(CONFIG_DIR, "config_rms.json") #### !!!! In ADC, wrong for this analysis !!!! + with open(rms_config_json, "r") as f: + rms_config_dict = json.load(f) + + kde_modality_function_parameters = rms_config_dict["kde_modality_function_parameters"] + skewness_function_parameters = rms_config_dict["skewness_function_parameters"] + report_vrms_function_parameters = rms_config_dict["report_vrms_function_parameters"] + + logger.info("Starting Vrms analysis for data read with dataProviderRNOG...") + vrms_arr, vrms_arr_force, vrms_arr_radiant0, vrms_arr_radiant1, vrms_arr_lt = calculate_vrms(trace_arr, event_info) + + logger.info(f"Number of RADIANT0 trigger events: {len(vrms_arr_radiant0[1])}, Number of RADIANT1 trigger events: {len(vrms_arr_radiant1[1])}, Number of LT trigger events: {len(vrms_arr_lt[1])}") + + logger.info(f"Calculating RMS (for monitoring.root) or Vrms (for dataProviderRNOG) modality and tail characteristics for each trigger type...") + modality_dict_force = kde_modality(vrms_arr_force, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_force = tail_fraction_and_trimmed_skew_two_sided(vrms_arr_force, all_channels, skewness_config=skewness_function_parameters) + if len(vrms_arr_force[1]) < 100: + logger.warning(f"FORCE trigger has less than 100 valid RMS (for monitoring.root) or Vrms (for dataProviderRNOG) entries ({len(vrms_arr_force[1])}). Results for the Vrms statistics may be unreliable.") + modality_force, tail_label_force = report_vrms_characteristics(modality_dict_force, tail_dict_force, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_radiant0 = kde_modality(vrms_arr_radiant0, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_radiant0 = tail_fraction_and_trimmed_skew_two_sided(vrms_arr_radiant0, all_channels, skewness_config=skewness_function_parameters) + if len(vrms_arr_radiant0[1]) < 100: + logger.warning(f"RADIANT0 trigger has less than 100 valid RMS (for monitoring.root) or Vrms (for dataProviderRNOG) entries ({len(vrms_arr_radiant0[1])}). Results for the Vrms statistics may be unreliable.") + modality_radiant0, tail_label_radiant0 = report_vrms_characteristics(modality_dict_radiant0, tail_dict_radiant0, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_radiant1 = kde_modality(vrms_arr_radiant1, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_radiant1 = tail_fraction_and_trimmed_skew_two_sided(vrms_arr_radiant1, all_channels, skewness_config=skewness_function_parameters) + if len(vrms_arr_radiant1[1]) < 100: + logger.warning(f"RADIANT1 trigger has less than 100 valid RMS (for monitoring.root) or Vrms (for dataProviderRNOG) entries ({len(vrms_arr_radiant1[1])}). Results for the Vrms statistics may be unreliable.") + modality_radiant1, tail_label_radiant1 = report_vrms_characteristics(modality_dict_radiant1, tail_dict_radiant1, all_channels, report_config=report_vrms_function_parameters) + + modality_dict_lt = kde_modality(vrms_arr_lt, all_channels, kde_modality_config=kde_modality_function_parameters) + tail_dict_lt = tail_fraction_and_trimmed_skew_two_sided(vrms_arr_lt, all_channels, skewness_config=skewness_function_parameters) + if len(vrms_arr_lt[1]) < 100: + logger.warning(f"LT trigger has less than 100 valid RMS (for monitoring.root) or Vrms (for dataProviderRNOG) entries ({len(vrms_arr_lt[1])}). Results for the Vrms statistics may be unreliable.") + + modality_lt, tail_label_lt = report_vrms_characteristics(modality_dict_lt, tail_dict_lt, all_channels, report_config=report_vrms_function_parameters) + plot_vrms_values_against_time(times, vrms_arr, all_channels, station_id, run_label, save_location, force_mask, radiant0_mask, radiant1_mask, lt_mask, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + plot_vrms_values_against_time_per_trigger(times, vrms_arr, all_channels, station_id, run_label, save_location, force_mask, radiant0_mask, radiant1_mask, lt_mask, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + + # Write detailed Vrms modality results to text files for each trigger type + write_vrms_modality_results(modality_force, tail_label_force, trigger_label="FORCE", station_id=station_id, run_label=run_label, results_dir=RESULTS_DIR, use_monitoring=use_monitoring) + write_vrms_modality_results(modality_radiant0, tail_label_radiant0, trigger_label="RADIANT0", station_id=station_id, run_label=run_label, results_dir=RESULTS_DIR, use_monitoring=use_monitoring) + write_vrms_modality_results(modality_radiant1, tail_label_radiant1, trigger_label="RADIANT1", station_id=station_id, run_label=run_label, results_dir=RESULTS_DIR, use_monitoring=use_monitoring) + write_vrms_modality_results(modality_lt, tail_label_lt, trigger_label="LT", station_id=station_id, run_label=run_label, results_dir=RESULTS_DIR, use_monitoring=use_monitoring) + + # The Vrms statistics can be misleading (especially for low event number) so the debugging plots are always generated + debug_plot_vrms_distribution(vrms_arr_force, modality_dict_force, channel_list=all_channels, station_id=station_id, run_label=run_label, trigger_label="FORCE", save_location=save_location, n_rows=12, n_cols=2, use_monitoring=use_monitoring) + debug_plot_vrms_distribution(vrms_arr_radiant0, modality_dict_radiant0, channel_list=all_channels, station_id=station_id, run_label=run_label, trigger_label="RADIANT0", save_location=save_location, n_rows=12, n_cols=2, use_monitoring=use_monitoring) + debug_plot_vrms_distribution(vrms_arr_radiant1, modality_dict_radiant1, channel_list=all_channels, station_id=station_id, run_label=run_label, trigger_label="RADIANT1", save_location=save_location, n_rows=12, n_cols=2, use_monitoring=use_monitoring) + debug_plot_vrms_distribution(vrms_arr_lt, modality_dict_lt, channel_list=all_channels, station_id=station_id, run_label=run_label, trigger_label="LT", save_location=save_location, n_rows=12, n_cols=2, use_monitoring=use_monitoring) + + # Vrms stability + reference_filename_rms = f"expected_{rms_label}/expected_{rms_label}_station{station_id}.json" + vrms_k_values, vrms_ref_mean, vrms_ref_std = load_values_json(REFERENCE_DIR, reference_filename_rms) + z_score_arr_vrms_force = calculate_z_score_parameter(vrms_arr_force, vrms_ref_mean, vrms_ref_std, all_channels) + flag_outliers_vrms_force = outlier_flag(z_score_arr_vrms_force, vrms_k_values, all_channels) + outlier_details_vrms_force = find_outlier_details(z_score_arr_vrms_force, vrms_k_values, flag_outliers_vrms_force, channel_list=all_channels, run_no=run_no_force, event_number=event_number_force) + write_vrms_outlier_details(outlier_details_vrms_force, station_id, run_label, trigger_label="FORCE", n_events=n_events_force, results_dir=RESULTS_DIR, use_monitoring=use_monitoring) + plot_vrms_values_against_time_single_trigger_zscore(times_force, vrms_arr_force, flag_outliers_vrms_force, z_score_arr_vrms_force, vrms_k_values, trigger_name = "FORCE", channel_list = all_channels, station_id=station_id, run_label=run_label, save_location=save_location, n_rows=12, n_cols=2, day_interval=day_interval, use_monitoring=use_monitoring) + + rms_arr_per_run_dict_force = get_rms_per_run(vrms_arr_force, run_no_force) + relative_median_shift_results = relative_median_shift(rms_arr_per_run_dict_force, all_channels) + with open(os.path.join(RESULTS_DIR, f"{rms_label}_relative_median_shift_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(relative_median_shift_results, f, indent=4) + create_heatmap_plot(relative_median_shift_results, label = "Relative Median Shift", save_dir = PLOTS_DIR, channel_list=all_channels, station_id = station_id,matrix_key = "median_shift_matrix", run_label=run_label, cmap="Reds") + rms_results = decision_metric(outlier_details_vrms_force, relative_median_shift_results, n_events_force=n_events_force, channels=all_channels) + with open(os.path.join(RESULTS_DIR, f"rms_stability_decision_results_force_trigger_station{station_id}_{run_label}.json"), "w") as f: + json.dump(rms_results, f, indent=4) + + ##### Glitching analysis - Same for both monitoring and dataProviderRNOG + logger.info("Starting glitching analysis...") + + # Load the configuration parameters for the glitching analysis from the JSON file + glitching_config_json = os.path.join(CONFIG_DIR, "config_glitching.json") + with open(glitching_config_json, "r") as f: + glitching_config_dict = json.load(f) + + config_glitching = glitching_config_dict["config_glitching_values"] + + glitch_info = binomtest_glitch_fraction(glitch_arr, all_channels, config_glitching=config_glitching) + write_glitching_results(glitch_info, station_id, run_label, all_channels, results_dir = RESULTS_DIR) + + glitching_violin_plot(glitch_arr, all_channels, station_id, run_label, save_location) + plot_glitch_q99_over_time(np.array(times), glitch_arr, all_channels, station_id, run_label, save_location) + + ##### Block offsets analysis + # Get the reference block offset results for the station + ref_block_offset_results_file = os.path.join(REFERENCE_DIR, "expected_block_offsets",f"expected_block_offsets_station{station_id}.json") + with open(ref_block_offset_results_file, "r") as f: + ref_block_offset_results = json.load(f) + + ##### Block offsets - dataProviderRNOG + + logger.info("Starting block offset analysis (dataProviderRNOG), results are not used to determine channel health, see warnings in the log file for channels with potential block offset issues. The block offsets are then removed.") + fit_block_offsets_before = get_block_offsets_before_removal(block_offsets_arr, event_info, all_channels) + fit_block_offsets_after = get_block_offsets_after_removal(trace_arr, event_info, all_channels, sampling_rate=sr) + + block_offset_stats = block_offset_statistics(fit_block_offsets_before, fit_block_offsets_after, all_channels) + block_offset_results_dict = write_block_offset_results(block_offset_stats, station_id, run_label, ref_block_off_dict = ref_block_offset_results, results_dir = RESULTS_DIR, use_monitoring=use_monitoring) + plot_block_offsets_violin_before_after_comparison(fit_block_offsets_before, fit_block_offsets_after, all_channels, station_id, run_label, save_location) + + # Debug plots + if args.debug_plot: + debug_plot_ratios(ratio_arr_dict=ratio_arr_dict, channels_order=channels_order, save_location=save_location, station_id=station_id, run_label=run_label, bins=30,) + debug_plot_snr_distribution(log_snr_arr, channel_list=all_channels, save_location=save_location, station_id=station_id, run_label=run_label, bins=30) + debug_plot_z_score_snr(z_score_arr_log_snr, channel_list=all_channels, save_location=save_location, station_id=station_id, run_label=run_label, bins=30) + + # Create summary CSV file + create_result_csv_file( + station_id, + run_label, + n_events_force, + surface_channels, + downward_channels, + upward_channels, + all_channels, + all_validation_results, + glitch_info, + block_offset_results_dict, + rms_results, + modality_dict_force, + modality_dict_lt, + modality_dict_radiant0, + modality_dict_radiant1, + outlier_details_snr, + CSV_DIR, + rms_label + ) + +