[Process] Establish a backend-neutral ML workflow with optional PyTorch support

[HongSik-Yun-Fusion]

Background

VAFT currently provides signal-processing and scientific-processing capabilities based primarily on NumPy/SciPy/scikit-learn, but it does not yet have a common API for neural-network or broader machine-learning workflows.

A prototype workflow developed for the 2026 VEST IRE anomaly-detection study provides a useful reference case. It already separates the workflow into reusable stages:

shot data
  -> preprocessing
  -> window / feature extraction
  -> shot-level dataset split
  -> model training
  -> anomaly scoring
  -> threshold calibration
  -> event composition
  -> event-level output

The prototype compares KNN, One-Class SVM, and an Autoencoder on a shared feature dataset; only the Autoencoder requires PyTorch. This suggests that VAFT should support ML in a backend-neutral way, with PyTorch as an optional backend rather than as a core scientific API dependency.

Note: This issue describes an initial architectural plan. Before implementation, review the existing VAFT architecture in detail and propose a different design if it provides cleaner ownership, reuse, or integration with the current processing, pipeline, and storage layers.

Goals

Add a minimal ML-processing infrastructure under vaft.process, tentatively centered on:

vaft.process
  |- features/          # reusable deterministic feature extraction where appropriate
  `- ml/
      |- dataset
      |- inference
      |- calibration
      |- sklearn/
      `- torch/         # optional backend

The design should:

• keep the public workflow backend-neutral;
• add PyTorch only as an optional dependency;
• separate deterministic feature extraction from model inference where practical;
• separate training from inference lifecycle;
• retain dataset, preprocessing, split, calibration, software, and model provenance;
• define stable artifact boundaries that can later integrate with the VAFT pipeline and FileDB/database layers;
• use the VEST IRE anomaly-detection workflow as the first reference implementation.

Ownership boundaries

vaft.process.ml should own computation such as:

• dataset preparation;
• model fitting;
• inference and scoring;
• threshold calibration;
• event-candidate composition.

It should not own:

• pipeline DAG orchestration;
• FileDB path policy;
• database registration;
• artifact replication;
• campaign scheduling.

Conceptually:

vaft.process.ml  -> computation
pipeline         -> orchestration
FileDB / DB      -> persistence

Optional PyTorch backend

PyTorch should not become a mandatory VAFT dependency. A tentative packaging direction is:

[project.optional-dependencies]
torch = [
    "torch..."
]

with usage such as:

pip install "vaft[torch]"

Without PyTorch installed, the following should still work:

import vaft
import vaft.process
import vaft.process.ml

Only PyTorch-specific functionality should raise an actionable optional-dependency error.

PyTorch types such as torch.Tensor, torch.nn.Module, and DataLoader should remain backend implementation details rather than required public scientific API types.

Common workflow contract

Start with a small contract rather than a large ML framework. Candidate concepts include:

FeatureSpec
FeatureDataset
SplitSpec

ModelSpec
ModelArtifact

InferenceResult
CalibrationResult

A target usage pattern could look like:

dataset = prepare_dataset(
    shots,
    feature_spec=...,
    split_spec=...,
)

model = train_model(
    dataset,
    model_spec=...,
)

result = infer(model, dataset)
calibration = calibrate_threshold(result, labels=...)
events = compose_events(result, calibration=calibration)

Exact names and module placement should be decided after architecture review.

Training and inference

Training and inference should be treated as separate lifecycle stages.

shot corpus
    -> feature dataset
    -> training
    -> model artifact
    -> per-shot inference
    -> scores / scientific products / event candidates

This separation is important for reproducibility, model reuse, and future pipeline integration.

Model artifact and provenance

A model artifact should contain or reference more than weights alone. At minimum, preserve:

• model specification / architecture;
• backend;
• feature schema;
• preprocessing configuration;
• training-shot selection;
• train/validation/test split;
• scaler / normalization;
• threshold calibration;
• VAFT/software version;
• evaluation metrics;
• model weights.

A future dataset/model fingerprint may be used to identify reproducible training configurations.

Reference implementation: VEST IRE anomaly detection

Port the existing IRE anomaly-detection prototype as the first reference workflow.

Current structure:

Stage 1:
    Ip + dIp/dt
        -> anomaly score

Stage 2:
    H-alpha + dH-alpha/dt + cross-signal features
        -> second anomaly score

both stages pass
    -> anomalous windows
    -> merged event-level candidates

Support the existing model families through the same dataset/inference contract:

• KNN;
• One-Class SVM;
• Autoencoder (optional PyTorch).

The dataset split must remain shot-level rather than window-level to avoid leakage.

Relation to physics-based event detection

ML output should not be interpreted directly as a physical reconnection onset.

Keep data-driven outputs such as:

anomaly score
anomalous window
IRE-like candidate interval

separate from physics-transparent observable events such as:

current_spike_onset
magnetic_burst_onset
H-alpha_burst_onset

This allows signal-processing and ML evidence to be compared rather than conflated. Final event candidates can later be connected to the common plasma-evolution timeline as one evidence producer with model/provenance metadata.

Initial phases

Phase 1 — ML core

• review current VAFT architecture and confirm or revise module placement;
• define vaft.process.ml ownership and minimal common types;
• define dataset/split and inference/calibration result contracts;
• add a reusable optional-dependency helper.

Phase 2 — optional PyTorch

• add an optional PyTorch extra;
• add vaft.process.ml.torch or an architecture-equivalent backend module;
• ensure VAFT imports and non-torch ML workflows remain usable without PyTorch;
• add a minimal CPU-only test.

Phase 3 — IRE reference workflow

• migrate the shared feature/dataset workflow;
• support KNN;
• support OCSVM;
• support Autoencoder when PyTorch is available;
• preserve shot-level splitting;
• support threshold calibration and event-level candidate composition;
• validate against the prototype workflow.

Phase 4 — artifact/provenance

• persist or serialize model specification and training metadata;
• preserve feature schema and preprocessing configuration;
• preserve dataset/split provenance and calibration metadata;
• define a stable ModelArtifact / InferenceResult boundary.

Phase 5 — future pipeline / DB integration

Follow-up work may connect these contracts to:

• batch/corpus-level training stages;
• shot-level inference stages;
• FileDB model-artifact storage;
• model registry metadata;
• database-scale inference products;
• event/state timeline integration.

These integrations do not need to be implemented in the first PR.

Non-goals

The initial implementation should not attempt to:

• make PyTorch mandatory;
• support every ML framework;
• build distributed/GPU training infrastructure;
• implement a full experiment-tracking or model-registry service;
• replace deterministic signal processing with neural networks;
• define ML anomaly peaks as physical reconnection onset;
• reimplement existing event detectors inside the ML layer;
• couple training and inference into one opaque API;
• lock FileDB/database schema before the processing contract is stable.

Acceptance criteria

• the architecture is reviewed before implementation and the proposed placement may be revised if a cleaner VAFT-native design is identified;
• PyTorch remains optional;
• VAFT and non-PyTorch ML functionality remain usable without PyTorch installed;
• the public ML contract is not tied to torch.Tensor or torch.nn.Module;
• shot-level dataset splitting is supported;
• deterministic feature extraction can remain separate from model inference;
• training and inference have separate lifecycle contracts;
• model/data/preprocessing/calibration provenance is retained;
• KNN, OCSVM, and optional Autoencoder can consume a shared reference dataset contract;
• ML inference can produce traceable event candidates without claiming physical onset semantics;
• the resulting artifact boundary is suitable for later pipeline/FileDB/database integration.

Long-term direction

This infrastructure should remain usable beyond IRE anomaly detection, including potential applications such as confinement-state classification, MHD-event classification, disruption/precursor inference, diagnostic reconstruction, equilibrium/stability/plasma-response surrogates, and time-series prediction.

The broader goal is to support a traceable workflow of:

scientific data
  -> features / representation
  -> data-driven model
  -> traceable scientific product

within VAFT without coupling the scientific API to one ML backend.