Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 0 additions & 13 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,6 @@ venv/
.venv/
.env

# contrail_ml and contrail_flights are excluded entirely — ML training/inference
# and the OpenSky real-traffic loader are separate concerns with their own heavy
# deps (scipy/sklearn/xgboost/mlflow; pyopensky/pandas/pyarrow). The server image
# has one job: run the CP-SAT gRPC service. Both run offline in their own envs.
contrail_ml/
contrail_flights/

# contrail_ml artifacts
mlruns/
data/processed/
*.joblib
*.npz

# Editor / OS noise and local notes
.DS_Store
*.ipynb_checkpoints/
Expand Down
21 changes: 0 additions & 21 deletions .env.example

This file was deleted.

36 changes: 3 additions & 33 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,41 +27,11 @@ jobs:
run: ruff check .

- name: Mypy (type-check)
run: mypy contrail_env service contrail_flights
run: mypy contrail_env service

- name: Pytest
run: pytest -q

ml:
# The contrail_ml ISSR model + MLOps lifecycle. Runs the HERMETIC ML tests
# (synthetic fallback only — no network, no credentials). Separate job so a
# heavy [ml] install never slows the core lint-type-test gate, and the ML
# deps never touch the runtime Docker image.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install package with ML + dev extras
run: |
python -m pip install --upgrade pip
pip install -e ".[ml,dev]"

- name: Generate gRPC stubs
run: bash scripts/gen_proto.sh

- name: Ruff (lint contrail_ml)
run: ruff check contrail_ml

- name: Mypy (type-check contrail_ml)
run: mypy contrail_ml

- name: Pytest (hermetic ML tests)
run: pytest tests/test_ml_*.py -q

docker-build:
runs-on: ubuntu-latest
steps:
Expand All @@ -88,8 +58,8 @@ jobs:
docker-publish:
runs-on: ubuntu-latest
# Only publish on pushes to main — not on pull requests.
# The full suite (core + ML) AND the image smoke-test must pass first.
needs: [lint-type-test, ml, docker-build]
# The full suite AND the image smoke-test must pass first.
needs: [lint-type-test, docker-build]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
permissions:
contents: read
Expand Down
9 changes: 2 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,5 @@ build/
# Generated gRPC stubs (regenerated by scripts/gen_proto.sh)
service/generated/

# contrail_ml — MLflow runs, versioned datasets, trained artifacts (all rebuilt)
mlruns/
data/processed/
*.joblib
*.npz
*.dvc
.dvc/
# Benchmark output
results.csv
10 changes: 4 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,10 @@ COPY pyproject.toml README.md ./
COPY contrail_env/ ./contrail_env/
COPY service/ ./service/
COPY scripts/ ./scripts/
# contrail_ml and the quantum solvers (pasqal_analog, xanadu_gbs) are
# intentionally excluded. This image has one job: run the CP-SAT gRPC server.
# ML training/inference and quantum sampling are separate concerns that carry
# heavy deps (scipy/sklearn/xgboost/pulser/strawberryfields) and run in their
# own environments. Including their source here would ship dead code that can
# never execute (deps absent) and would crash at request time if called.
# Only contrail_env + service ship here — the image's one job is the CP-SAT gRPC
# server. The quantum solver modules (pasqal_analog, xanadu_gbs) ride along in
# contrail_env and run on their built-in fallbacks; the optional [quantum] SDKs
# (pulser/strawberryfields) are not installed, keeping the runtime image lean.
RUN pip install .

# The gRPC stubs are gitignored — generate them from solver.proto at build time.
Expand Down
54 changes: 11 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ The dashboard has six tabs: live CP-SAT convergence (over ZMQ), the conflict-gra
topology, QUBO matrix statistics (size, sparsity, penalty constants), the
chosen-option trade-offs, the quantum benchmark (CP-SAT vs Pasqal vs Xanadu over
N seeds, with live convergence curves for the BO loop and the GBS sampler), and a
geographic map — the predicted ISSR risk as a marker overlay on a real Plotly
geographic map — the ISSR risk as a marker overlay on a real Plotly
`geo` basemap (country borders / coastlines, drawn with SVG and bundled offline
vectors, so it needs no WebGL or network) with the chosen vs context routes on top.

Expand All @@ -94,51 +94,24 @@ The benchmark also runs headless:
python -m contrail_env.benchmark --flights 4 --seeds 5 --csv results.csv
```

## ISSR model — real weather instead of synthetic blobs
## ISSR field

By default the airspace's contrail zones are synthetic Gaussian blobs. The
`contrail_ml` package replaces them with a trained model that predicts
ice-supersaturated regions (ISSRs) from real weather, bias-corrected against
in-situ IAGOS humidity, and wraps it in a full MLOps lifecycle (versioned data →
train → calibrate → register with MLflow → serve → monitor).

The model plugs in through the **same `ISSRField` interface** the synthetic
field uses, so `World` and the QUBO assembly are untouched — you flip a switch:

```python
from contrail_env import default_european_world
world = default_european_world(issr_source="ml", issr_kwargs={...}) # vs "synthetic"
```

Over gRPC the `ScenarioConfig` gained an `issr_source` field (default
`"synthetic"`, so existing clients are unaffected). Install the extra and try the
whole serving seam offline:

```
pip install -e ".[ml]"
python -m contrail_ml serve-check # ML ISSR field -> CP-SAT solve
python -m contrail_ml train --synthetic --no-mlflow # CV, fit, calibrate, baseline table
```

The science, the model, and how to read the model-vs-baselines table are in
[docs/ML.md](docs/ML.md); data sources (IAGOS / ARCO-ERA5 / GFS) in
[docs/DATA.md](docs/DATA.md). The hermetic tests use a guarded synthetic
fallback — no network, no credentials.
The airspace's contrail zones (ice-supersaturated regions) are synthetic
Gaussian blobs — a controllable obstacle field for the optimizer to route
around. The field is consumed through a small `ISSRField` interface
(`rhi_excess`, `is_inside`, `mask_grid`), so the *source* of the field is
pluggable without touching `World` or the QUBO assembly.

## Layout

```
contrail_env/ synthetic environment, QUBO assembly, CP-SAT solver,
quantum pipelines (pasqal_analog, xanadu_gbs, quantum_common,
bayes_opt) and the benchmark protocol (benchmark.py)
contrail_ml/ ISSR model (features, RHiCorrector, calibration), MLflow
registry, serving (MLIssrField), monitoring, and the data
loaders (IAGOS/ERA5/GFS) — the [ml] extra
contrail_env/ synthetic environment, ISSR field, geo anchor, QUBO assembly,
CP-SAT solver, quantum pipelines (pasqal_analog, xanadu_gbs,
quantum_common, bayes_opt) and the benchmark protocol (benchmark.py)
service/ gRPC service, ZMQ progress streaming, client
gui/ PyQt6 dashboard
tests/ environment build, CP-SAT vs brute-force, quantum solvers vs
brute-force, benchmark round-trip, gRPC round-trip, and the
hermetic contrail_ml suite (test_ml_*.py)
brute-force, benchmark round-trip, gRPC round-trip, GUI map panel
```

## Development
Expand All @@ -154,11 +127,6 @@ CI runs the same checks on every push and pull request.

## Roadmap

- **Done:** real ISSR model (`contrail_ml`) replacing the synthetic field, with
the train → calibrate → register → serve → monitor MLOps lifecycle (see
[docs/ML.md](docs/ML.md)). Next: run it end-to-end on real IAGOS + ERA5 (the
loaders are written; they need portal registration) and report the honest
real-data comparison table.
- Run the Pasqal pipeline on real Pulser hardware: needs `[quantum]` extras plus a
conflict graph that embeds as a valid unit-disk register (auto-detected; the
built-in simulator is the fallback).
Expand Down
49 changes: 23 additions & 26 deletions contrail_flights/geo.py → contrail_env/geo.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,13 @@
"""
geo.py — local sim frame <-> real (lon, lat), for the real-flight loader.
geo.py — Where the synthetic world sits on Earth (local sim frame <-> lon/lat).

This is a DELIBERATE, independent mirror of
`contrail_ml.features.GeoAnchor`: the brief requires `contrail_flights` to be
a standalone sibling of `contrail_ml` (no import between the two), so the
transform is duplicated here rather than shared. The two MUST stay in sync —
same formula, same default origin — so the predicted ISSR field (placed via
the contrail_ml anchor) and the real flights (placed via this anchor) agree on
where things are on the map. `tests/test_geo_transform_roundtrip.py` checks the
inverse, and a cross-check test guards against drift from the contrail_ml copy.
The optimizer works entirely in the local 1500x800 km Cartesian box; geography
only matters for the GUI Map tab, which needs to place that box on a real
basemap. This module owns that one mapping so the map has a home for it without
reaching into any heavier package.

lat = origin_lat + y_km / KM_PER_DEG_LAT
lon = origin_lon + x_km / (KM_PER_DEG_LAT * cos(lat))

Only numpy is needed, so this module imports cleanly in the lean install (the
[flights] extra is only for the actual OpenSky pull, not the geometry).
"""

from __future__ import annotations
Expand All @@ -23,20 +16,26 @@

import numpy as np

# Must match contrail_ml.config.MLConfig defaults so flights and the predicted
# field share one coordinate system.
DEFAULT_ORIGIN_LAT = 43.0
DEFAULT_ORIGIN_LON = -5.0
KM_PER_DEG_LAT = 111.0

def _maybe_scalar(arr: np.ndarray, like: np.ndarray | float) -> np.ndarray | float:
"""Return a Python float when the input was scalar, else the array."""
if np.isscalar(like) or (isinstance(like, np.ndarray) and like.ndim == 0):
return float(arr)
return arr


@dataclass(frozen=True)
class GeoAnchor:
"""Maps the local sim frame (x_km east, y_km north) to/from (lon, lat)."""
"""Maps the local sim frame (x_km east, y_km north) to geography.

origin_lat: float = DEFAULT_ORIGIN_LAT
origin_lon: float = DEFAULT_ORIGIN_LON
km_per_deg_lat: float = KM_PER_DEG_LAT
The sim is a flat Cartesian box; the map lives on (lon, lat). One
small-angle anchor ties them together so the risk overlay and the routes
render over the same place.
"""

origin_lat: float = 43.0
origin_lon: float = -5.0
km_per_deg_lat: float = 111.0

def local_to_geo(
self, x_km: np.ndarray | float, y_km: np.ndarray | float
Expand All @@ -59,8 +58,6 @@ def geo_to_local(
return _maybe_scalar(x, lon_deg), _maybe_scalar(y, lat_deg)


def _maybe_scalar(arr: np.ndarray, like: np.ndarray | float) -> np.ndarray | float:
"""Return a Python float when the input was scalar, else the array."""
if np.isscalar(like) or (isinstance(like, np.ndarray) and like.ndim == 0):
return float(arr)
return arr
# The canonical anchor: places the 1500x800 km box over south-west -> central
# Europe (roughly Madrid to Frankfurt), matching the default world geometry.
EUROPEAN_ANCHOR = GeoAnchor(origin_lat=43.0, origin_lon=-5.0)
17 changes: 3 additions & 14 deletions contrail_env/world.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,14 +178,8 @@ def default_european_world(
- Uniform 3x3 sector grid with capacity 3

issr_source:
"synthetic" (default) — the random Gaussian-blob ISSR field, exactly
as before, so all existing callers/tests are unaffected.
"ml" — a trained model's predicted field (contrail_ml.MLIssrField),
which honours the SAME ISSRField interface, so World/qubo need no
changes. contrail_ml is imported LAZILY here, so the core install
(no [ml] extra) still imports contrail_env fine as long as the ml
source is not requested.
`issr_kwargs` is forwarded to the chosen ISSR-field factory.
"synthetic" (default) — the random Gaussian-blob ISSR field.
`issr_kwargs` is forwarded to the ISSR-field factory.
"""
from .synthetic_issr import random_issr_field

Expand All @@ -205,14 +199,9 @@ def default_european_world(
seed=seed,
**kw,
)
elif issr_source == "ml":
# Lazy import: only the "ml" branch pulls in the [ml] extra.
from contrail_ml.issr_field import ml_issr_field

issr = ml_issr_field(**kw)
else:
raise ValueError(
f"unknown issr_source {issr_source!r} (use 'synthetic' or 'ml')"
f"unknown issr_source {issr_source!r} (use 'synthetic')"
)

from .airspace import uniform_sector_grid
Expand Down
46 changes: 0 additions & 46 deletions contrail_flights/__init__.py

This file was deleted.

Loading
Loading