Develop 2.0

[SteSeg]

Disclaimer

This PR mainly proposes better tally management and more consistent across the different apps. Moreover, it proposes major new implementations like more benchmark quality metrics other than classic C/E and automatic report generation. It is an ensamble of multiple PR that were merged on a the develop_2.0 branch of my fork that caused it to be consistently ahead of this default branch. Here below the summary of all the new features.

Self consistency part 1:

Replace DataFrame-based tally conversion with nD OFB tallies

Summary

This PR removes the intermediate pandas.DataFrame step when converting OpenMC tallies from a statepoint file to benchmark_results.h5.

Previously, tallies were converted as:

OpenMC tally -> pandas.DataFrame -> xarray.DataArray -> HDF5

Because DataFrames are inherently 2D, this imposed shape limitations and made it harder to preserve full tally structure.

With this PR, tallies are handled as true multi-dimensional arrays, consistent with OpenMC and with the OFB specification model.

What changes

• Introduced BaseTally and Tally objects for OFB tally handling.
• Added src/openmc_fusion_benchmarks/backends/ to support transport-code-specific conversions.
• Implemented OpenMC backend conversion in src/openmc_fusion_benchmarks/backends/openmc/.
• Updated result handling so dimensions are preserved according to the benchmark specification:
  • score
  • nuclide
  • filters
  • and any additional dimensions required by the spec.

Why this matters

• Preserves physical/data structure of tallies end-to-end.
• Removes artificial 2D constraints from the conversion pipeline.
• Establishes a backend abstraction for future transport codes.
• Aligns benchmark_results.h5 tally representation with OFB specification semantics.

Usage example

import openmc_fusion_benchmarks as ofb

# Load OFB benchmark results
br = ofb.BenchmarkResults("benchmark_results.h5")

# List available tallies
print(br.tallies)

# Retrieve one tally by name
my_tally = br.get_tally(name="mytally")

# Access mean values (xarray-backed)
mean = my_tally.mean

# Inspect shape
print(mean.shape)

Notes for reviewers

This PR is part of a stacked series.
Related PRs in this series use the [Self consistency] prefix.

Part 2

this PR adds the internal piping to check whether resulting tallies from a benchmark (here provided with the openmc backend), converted to an ofb Tally, has the shape consistent with what defined in the benchmark specifications file.

In addition, a get_spec_consistency_report() method has been implemented in the BenchmarkResults/Results class. Such report provides informations about the actual consistency of all the tallies resulting from a given benchmark run with the specifications requirements. Example usage:

import openmc_fusion_benchmarks as ofb

# Get results from a benchmark run
br = ofb.BenchmarkResults(`benchmark_results.h5`)

# Get tallies consistency report
report = br.get_spec_consistency_report()
# Get only the tally results-specifications mismatches 
mismatches = br.get_spec_consistency_report(only_mismatches=True)

# Print
print(report)
print(mismatches)

Part 3

Now that the openmc_fusion_benchmark's Tally object is a multi-dimensional array (which has its own convention for dimensions), it may become confusing to navigate all the dimensions: there are new dimensions for every filter, score, nuclides etc. (and with the adaptation of the TMCManager uncertainty quantification method there will be dimension also for perturbations and realizations). Hence, we implemented here the possibility of printing a tally_dimension_report for easy inspection.
Example usage:

import openmc_fusion_benchmarks as ofb

# Get benchmark results file
br = ofb.BenchmarkResults('benchmark_results.h5')

# Extract tally
tally = br.get_tally('tally_name')

# print tally dimension report
print(tally.format_dimension_report())

It should print something like this:

Tally neutron_leakage (id=1)
OFB dims            : ['realization', 'particle', 'surface', 'energy', 'nuclide', 'score']
OFB shape           : (1, 1, 1, 134, 1, 1)
TMC axes            : realization=1
Filter axes         : particle=1, surface=1, energy=134
Nuclide/score sizes : nuclide=1, score=1
OpenMC raw equivalent: (134, 1, 1)  (flat_filters, nuclide, score)

As always, we provide the code for the openmc backend. Code for other backends (e.g. serpent, 'mcnp` etc.) is always welcome.

Part 4

Implemented the tmc_mode metadata (sequential, matrix or diagonal) in the TMCStatePoint object.

import openmc_fusion_benchmarks as ofb

tsp = ofb.TMCStatePoint('path/to/tmc_statepoint.500.h5')
print(tsp.tmc_mode)

Part 5: report generation

Summary

This PR adds a modular report-generation scaffold for OFB benchmark results. It provides the raw structure (metadata + sources + plots) and supports YAML/PDF output. Scoring integration is intentionally left for a later PR.

What’s Included

• Report models and builder
• Absolute + C/E plots with experimental 3σ bands
• YAML renderer (machine-readable)
• PDF renderer (human-readable)
• Automatic log/linear scale (default: log if range > 100x)

Architecture

• report/models.py
  • ReportMetadata, ResultSource, ReportConfig, PlotStyle, PlotSpec, Report
• report/builder.py
  • build_report(...)
• report/plots.py
  • plotting + auto-scale
• report/renderers.py
  • render_yaml(...), render_pdf(...), render_plots_for_report(...)
• report/init.py
  • public API exports

Usage

from pathlib import Path
from openmc_fusion_benchmarks.benchmark_results import BenchmarkResults
from openmc_fusion_benchmarks.report import (
    ReportConfig,
    ReportMetadata,
    ResultSource,
    build_report,
    render_yaml,
    render_pdf,
)

exp = BenchmarkResults.from_file("reference_results.h5")
calc = BenchmarkResults.from_file("benchmark_results.h5")

metadata = ReportMetadata(
    title="FNG validation report",
    benchmark_id="oktavian_al",
    description="Short experiment description",
    model_description="Short model description",
    code_name="openmc",
    code_version="0.15.0",
)

sources = [
    ResultSource(name="experiment", kind="experiment", results=exp),
    ResultSource(name="calculation", kind="calculation", results=calc),
]

config = ReportConfig(output_dir=Path("report_out"), include_yaml=True, include_pdf=True)

report = build_report(metadata, sources, config)
render_yaml(report, config.output_dir / "report.yaml")
render_pdf(report, config.output_dir / "report.pdf", config.output_dir / "plots")

Configuration knobs

• ReportMetadata: title, benchmark ID, model description, code name/version, notes
• ResultSource: experiment/calculation/database inputs
• PlotStyle:
  • y_scale="auto" by default (log if range > 100x)
  • override with "linear"/"log"
  • labels and titles
• ReportConfig:
  • plot_tallies to limit plots
  • include_yaml / include_pdf toggles

Notes / limitations

• This is a scaffold only; scoring integration will follow later.

part 6: Benchmark quality values beyond C/E

Summary

This PR introduces a first end-to-end validation framework for comparing benchmark results against reference data. The pipeline goes from per-point metrics to observable (tally) aggregates and then to benchmark-level aggregates. It also includes a grading/qualitative status infrastructure (OK/WARNING/OUTLIER, ACCEPTABLE/BORDERLINE/PROBLEMATIC, dashboard score), but that grading output is disabled by default for now and kept as a skeleton for future use.

Rationale and structure

The goal is a consistent, repeatable workflow to evaluate calculations vs reference data while preserving full quantitative metrics. The grading layer is intentionally kept off for external presentation until we finalize thresholds and interpretation.

Scoring flow

1. A comparison point represents one tally value (e.g., energy bin, foil value, leakage value).
2. Point-level metrics are computed.
3. Points are aggregated into a single observable (tally).
4. Observables are aggregated into a benchmark comparison.

Point-level metrics (quantitative)

For calculated value C, experimental/reference value E, and uncertainties u_C, u_E:

• C/E
• Relative deviation: (C - E) / E
• Absolute deviation: |C - E|
• Combined uncertainty: sqrt(u_E^2 + u_C^2)
• Normalized residual: (C - E) / sqrt(u_E^2 + u_C^2)
• chi2 contribution: (C - E)^2 / (u_E^2 + u_C^2)

Observable-level metrics (quantitative)

• mean_bias
• mean_abs_relative_deviation
• rms_relative_deviation
• mean_abs_normalized_residual
• reduced_chi2
• fraction_within_1sigma / 2sigma / 3sigma
• pass_count / warning_count / outlier_count

Benchmark-level metrics (quantitative)

• weighted_mean_bias
• weighted_rms_relative_deviation
• global_reduced_chi2
• total_point_count
• outlier_fraction

Qualitative grading (skeleton only; disabled by default)

• Point status: OK / WARNING / OUTLIER
• Benchmark status: ACCEPTABLE / BORDERLINE / PROBLEMATIC
• dashboard_score (0-100)

These are not surfaced by default in comparison outputs. They can be enabled explicitly via include_grading=True.

Usage

Basic workflow (benchmark-level)

from openmc_fusion_benchmarks.validation import compare_benchmark_results
from openmc_fusion_benchmarks.benchmark_results import BenchmarkResults

reference = BenchmarkResults.from_file("reference_results.h5")
candidate = BenchmarkResults.from_file("benchmark_results.h5")

bench = compare_benchmark_results(
    benchmark_id="oktavian_al",
    reference_source="experiment",
    reference=reference,
    candidate=candidate,
    # optional:
    tally_names=["tally_1", "tally_2"],
    observable_type_map={
        "tally_1": "spectrum",
        "tally_2": "reaction_rate",
    },
    flatten_dims_map={
        "tally_1": ["energy", "surface", "nuclide", "score"],
        "tally_2": ["energy", "nuclide", "score"],
    },
)

print(bench.weighted_rms_relative_deviation, bench.global_reduced_chi2, bench.outlier_fraction)

Enabling grading output explicitly

bench = compare_benchmark_results(
    benchmark_id="oktavian_al",
    reference_source="experiment",
    reference=reference,
    candidate=candidate,
    include_grading=True,
)

print(bench.benchmark_status, bench.dashboard_score)

Observable-level (tally-level)

from openmc_fusion_benchmarks.validation import compare_tallies

obs = compare_tallies(
    observable_name="tally_1",
    observable_type="spectrum",
    reference=exp_tally,
    candidate=calc_tally,
    flatten_dims=["energy", "nuclide", "score"],
)

Notes

• Grading/status fields are present but suppressed by default so only quantitative metrics surface in normal usage.
• code_name and code_version are optional and are pulled from run metadata when available.

Part 7: Report generation made automatic and API exposed

Summary

This PR introduces an end-to-end report generation API for benchmarks, including optional report creation directly from Benchmark.run(). The report workflow builds structured report metadata, renders plots (absolute + C/E), and adds validation quality plots and observable-level summary charts based on verbosity. Reports can be rendered to YAML and PDF.

What is new

• Benchmark.run(..., generate_report=False, report_config=None) now optionally generates a report after a benchmark run.
• Default reference results are pulled from results_database/{benchmark_name}/experiment.h5 when available.
• A complete reporting pipeline:
  • Build report data from ResultSource entries.
  • Render YAML and PDF outputs.
  • Add plot artifacts and PDF pages for absolute, C/E, quality metrics, and observable summaries.

API overview

1) Run-time report generation

bench.run(generate_report=True)

This triggers report generation after the run completes. If report_config is omitted, defaults are:

• output_dir=report/
• include_yaml=True
• include_pdf=True
• verbosity=2

2) Manual report pipeline

Use this when you want to customize the report or run it outside Benchmark.run().

from pathlib import Path
from openmc_fusion_benchmarks.benchmark_results import BenchmarkResults
from openmc_fusion_benchmarks.report import ReportConfig, ResultSource, build_report, render_pdf, render_yaml

reference = BenchmarkResults.from_file("reference_results.h5")
candidate = BenchmarkResults.from_file("benchmark_results.h5")

sources = [
    ResultSource(name="reference", kind="experiment", results=reference),
    ResultSource(name="calculation", kind="calculation", results=candidate),
]

config = ReportConfig(
    output_dir=Path("report"),
    include_yaml=True,
    include_pdf=True,
    verbosity=2,
)

report = build_report(sources, config)
render_yaml(report, config.output_dir / "report.yaml")
render_pdf(report, config.output_dir / "report.pdf", config.output_dir / "plots")

3) Config objects

• ReportConfig:

  • output_dir: output directory root.
  • include_yaml: enable YAML output.
  • include_pdf: enable PDF output.
  • plot_tallies: optional list of tallies to plot.
  • verbosity: controls level of detail for plots/sections.

• ResultSource:

  • name: display name.
  • kind: typically experiment (reference) or calculation (candidate).
  • results: a BenchmarkResults instance.
  • tally_names: optional list of tallies to include.

Workflow details

Build report data

build_report() collects:

• benchmark metadata from specifications and run metadata
• source metadata (file paths, tally list)
• plot definitions for each selected tally

Render outputs

• render_yaml() writes a machine-readable report summary.
• render_pdf() composes a full PDF report:
  • Summary page (benchmark title, reference, validation case)
  • Optional specification summary (verbosity > 0)
  • Per-tally plots for absolute values + C/E
  • Observable/tally-level summary charts
  • Quality evaluation plots (validation metrics)

Validation quality plots (verbosity-driven)

A dedicated "Quality evaluation" section is added with per-point metrics. Metrics included depend on verbosity:

• Verbosity 0: C/E
• Verbosity 1: C/E, chi2 contribution
• Verbosity 2 (default): C/E, relative deviation, combined uncertainty, chi2 contribution
• Verbosity 3: C/E, relative deviation, absolute deviation, combined uncertainty, normalized residual, chi2 contribution

Each plot includes the mathematical expression and a short description of the metric.

Observable summary charts

A dedicated "Observable summary" page aggregates per-tally metrics into bar charts. Metrics depend on verbosity:

• Verbosity 0: rms_relative_deviation
• Verbosity 1: rms_relative_deviation, reduced_chi2
• Verbosity 2: mean_bias, mean_abs_relative_deviation, rms_relative_deviation, reduced_chi2
• Verbosity 3: adds mean_abs_normalized_residual

Notes

• The report uses calculation results from benchmark_results.h5 and reference results from experiment.h5 if available.
• Grading/status metrics are intentionally disabled in report output; only quantitative metrics are shown.
• Report generation is non-blocking for workflows that do not require reporting (default is off).