diff --git a/docs/STATUS.md b/docs/STATUS.md index b4ff05b..378b77f 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -3,9 +3,10 @@ Read this first in every session, then `CLAUDE.md` (once it exists) and the ADRs relevant to your task. Keep it short; history lives in git. -**Last update:** 2026-10-17 · **Phase:** 4 in progress – PR 4-A contracts, PR 4-B storage; Phase 3 (3-A -`phase-3/a-safety`, 3-B `phase-3/b-controller`, 3-C `phase-3/c-ui`) implemented, waiting for the -maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/zealous-mayer-j5oda9`) +**Last update:** 2026-10-19 · **Phase:** 4 in progress – PR 4-A contracts, PR 4-B storage, PR 4-D PV +forecast; Phase 3 (3-A `phase-3/a-safety`, 3-B `phase-3/b-controller`, 3-C `phase-3/c-ui`) +implemented, waiting for the maintainer's shadow test · **Branch:** `phase-4/d-pv-forecast` +(stacked on `phase-4/b-storage`) ## Done @@ -58,6 +59,44 @@ maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/ - Follow-ups: `size_report` is not on a diagnostics page yet; a too-new `timeseries.db` is only logged (a UI hint follows in 4-H). +- **Phase 4 – PR 4-D (WP5, PV forecast; branch `phase-4/d-pv-forecast`, stacked on 4-B)**: + - Part A – adapters: template-driven sources (`suncore/forecast/pv/`) for Solcast + (`detailedForecast` attributes, fallback read-only action `solcast_solar.query_forecast_data`, + never the Solcast API), Forecast.Solar (core, `energy/solar_forecast`, `config_entry_id` when + several entries are linked), Open-Meteo Solar Forecast (`wh_period_15m`) and the generic + attribute template; pure parsing onto the source's native grid (DST-safe, unknown slots stay + `None`). Refresh service: every 30 min and on entity changes (debounced), keeps the last good + data, bounded gap filling, stores issues in `timeseries.db` hourly or on a material change + (≥ 15 %). Nightly accuracy bookkeeping (MAE and bias = actual − forecast per source and + horizon bucket 0–6/6–24/24–48 h over 14 days, daylight quarter hours only, clipped ones + excluded, `learned_param` `forecast_accuracy`). + `GET /api/v1/forecast/pv` and `/forecast/accuracy`; the demo publishes three deterministic + biased sources. + - Part B – blending, calibration, clipping: `blend.py` (weights per horizon bucket from the + inverse 14-day MAE, minimum weight 0.1, manual weights as shares, equal weights until every + auto source has ≥ 96 scored quarter hours; renormalised per slot; P50 weighted, P10/P90 from + the sources with quantiles widened by the sources' disagreement); `calibration.py` + (multiplicative factor per local hour × month-group season × weather class, bounded 0.5–1.5, + shrunk with `n/(n+16)`, learnt nightly per source from the day-start issue without clipped + slots, applied only after `min_history_days` = 30 days with data, stored in `forecast_bias`); + `clearsky.py` (weather class = clear-sky index of the forecast itself: sun position + Haurwitz + clear sky on the configured arrays with HA's `latitude`/`longitude` from `get_config`; + fallback without location or `kwp`: the day's forecast peak against the 95th percentile of + the stored daily peaks); + `clipping.py` (grid export at the export limit while the forecast is higher; an inverter + plateau derived from the data – no AC-rating field exists, none was invented); `learning.py` + (nightly order clipping → accuracy incl. `blended` → bias tables). The service blends on the + 15-min grid for now → +48 h, stores the blend as source `blended`, and the API shows the + weights, the calibration state per source and the blend without and with the learnt + calibration. The demo site now declares its 8 kWp horizontal plant. + - Deviation: the built-in PV fallback (Open-Meteo API, question 5) is **not shipped**; it moves + to Phase 6 with the price fallbacks (Open-Meteo terms and live azimuth convention + `UNVERIFIED`, plan §"Deviations during implementation"). + - Zero-export plants (`export_limit_w` 0 or 0 %): clipping by the export limit is not detected + (grid ≈ 0 W also without curtailment); only the inverter-plateau rule applies. + - Follow-ups: the weights come from the *uncalibrated* sources' MAE (a calibrated source is + under-weighted until a calibrated accuracy is scored); no UI yet (4-H). + - **Fix after the first live test (2026-10-16, branch `fix/device-freshness`)**: on the maintainer's HA the SunSpec battery SoC (84 % for 25 min) and the `smartthinq_sensors` power sensors (5 W for a day) never advance `last_reported` for an unchanged value, so SunCore marked them "stale" and the @@ -548,6 +587,11 @@ maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/ | Default sensor cadences (power sensor 60 s = the client resync, energy counter without `counter_update_s` 300 s); whether a quiet plug re-reports an unchanged value so that `last_reported` moves | `suncore/app/cadence.py`, `research/ha-websocket-api.md` | shadow-mode run on the maintainer's system; set `power_update_s`/`counter_update_s` where a sensor is slower (`power_update_s` is in the device editor since WP10) | | Controller tuning (deficit delay/budget, downside spread of the smoothing, counter-only climate needs `nominal + span + hysteresis` free surplus to shift; battery share without a goal = battery first) | `suncore/controller/`, `tests/controller/test_climate_equivalence.py` | shadow-mode run on the maintainer's system (plan exit criterion) | | evcc REST API paths | `research/competitors.md` | Phase 6, MQTT API is verified and sufficient | +| Solcast day sensors d3/d4: translation keys `total_kwh_forecast_d3`/`_d4` from the research notes; the date-time format the fallback action `query_forecast_data` accepts for `start_date_time`/`end_date_time` (SunCore sends ISO 8601 with offset) | `templates/forecast/solcast.pv.yaml`, `suncore/forecast/pv/adapter.py` | maintainer's HA with Solcast | +| Open-Meteo Solar Forecast: translation key assumed equal to the description key; CC BY 4.0 attribution still to be shown in the UI (4-H) | `templates/forecast/open_meteo_solar_forecast.pv.yaml` | a running integration; PR 4-H | +| Forecast.Solar `wh_hours`/`watt_hours_period` keys read as period *start* (vendor API unreachable) | `templates/forecast/forecast_solar.pv.yaml`, `suncore/forecast/pv/parsing.py` | live comparison with the integration's sensors | +| Supervisor proxy forwards `energy/solar_forecast` | `suncore/ha/` (`fetch_solar_forecast`) | first run on HA OS | +| PV calibration constants (classes kt 0.3/0.65, shrink k = 16, decay 0.98, factor 0.5–1.5) and the clipping thresholds (plateau ±0.3 % for ≥ 1 h on ≥ 2 days; export within max(3 %, 50 W)) are reasoned, not fitted to real data | `suncore/forecast/pv/` | 30+ days of the maintainer's data | | Solve times on Raspberry Pi 4 | ADR-0003 | Phase 4 benchmark | ## Environment notes diff --git a/docs/plans/phase-4.md b/docs/plans/phase-4.md index 0fed4f4..d0a850d 100644 --- a/docs/plans/phase-4.md +++ b/docs/plans/phase-4.md @@ -57,7 +57,8 @@ Phase 4 is done when (roadmap Phase 4 milestones plus the concept): 1. **Forecast adapters:** Solcast (P10/P50/P90 from the attributes or the read-only query action, never the Solcast API), Forecast.Solar (core, via `energy/solar_forecast`), Open-Meteo Solar Forecast (HACS), a generic attribute adapter (mapping per template), and the built-in fallback - behind an explicit opt-in (question 5). Several sources are blended by measured skill; a bias + behind an explicit opt-in (question 5; deferred to Phase 6, see "Deviations during + implementation"). Several sources are blended by measured skill; a bias calibration per hour of day × season × weather class (cloudiness) is learnt; clipping (inverter limit, export limit) is recognised and excluded from the statistics; accuracy (MAE, bias per source and horizon) is stored and shown. @@ -150,10 +151,11 @@ region; fixtures from verified shapes; templates with `source` and a conformance ### WP5 – PV forecast · `device-adapters` (adapters), `optimizer-engineer` (Opus, blending/calibration) `suncore/forecast/pv/`: Solcast, Forecast.Solar, Open-Meteo Solar Forecast, generic attribute, -built-in fallback (question 5). Blending by inverse MAE over 14 days with a minimum weight and a -manual override; bias calibration per hour × season × weather class (clear-sky index buckets -from the forecast itself), bounded and shrinking to 1 with little data, active after -`min_history_days`; clipping detection; accuracy in `timeseries.db`. +built-in fallback (question 5; deferred to Phase 6, see "Deviations during implementation"). +Blending by inverse MAE over 14 days with a minimum weight and a manual override; bias +calibration per hour × season × weather class (clear-sky index buckets from the forecast +itself), bounded and shrinking to 1 with little data, active after `min_history_days`; +clipping detection; accuracy in `timeseries.db`. ### WP6 – Load forecast · `optimizer-engineer` (Opus) @@ -400,7 +402,12 @@ shadow and assisted run is the acceptance test. Phase 6). - Open-Meteo API terms are known only from the repository README (free for open-source and non-commercial use, fair use, 10 000 requests/day, CC BY 4.0 attribution); the separate terms - page was unreachable. The built-in PV fallback stays opt-in and off. + page was unreachable. +- The built-in PV fallback (Open-Meteo API) is **not shipped in Phase 4** either (PR 4-D): its + terms and the azimuth convention of a live response are `UNVERIFIED` (question 5 default). It + moves to Phase 6 together with the price fallbacks, opt-in and off as planned; Phase 4 reads PV + forecasts only from the integrations the user already runs (Solcast, Forecast.Solar, + Open-Meteo Solar Forecast, generic template). - `BatteryAction` is part of the 4-A contracts but rejected by the gate until WP10 (ADR-0011 addendum); its semantics are proposed in ADR-0013. diff --git a/docs/research/ha-websocket-api.md b/docs/research/ha-websocket-api.md index 72f6aed..0c340f3 100644 --- a/docs/research/ha-websocket-api.md +++ b/docs/research/ha-websocket-api.md @@ -49,7 +49,7 @@ the connector takes the URL and token as parameters. | `call_service` | `domain`, `service`, `service_data`, `target`, `return_response` | actuation; `return_response: true` for actions that return data (`weather.get_forecasts`, `tibber.get_prices`, `solcast_solar.query_forecast_data`, `nordpool.get_prices_for_date`) | | `get_states` | – | initial snapshot | | `get_services` | – | capability discovery (which actions exist) | -| `get_config` | – | unit system, time zone, currency, location, version | +| `get_config` | – | unit system, time zone, currency, location, version; `latitude`/`longitude` (floats, degrees) feed the PV clear-sky model of the forecast calibration – verified in `home-assistant/core` `5cfe9aad7cb428c2c2ebb449719ae81976780de1` `homeassistant/core_config.py` `Config.as_dict` (lines 706, 713) returned unchanged by `websocket_api/commands.py` `handle_get_config` (only `external_url` is removed for local-only users) | | `config/entity_registry/list_for_display` (preferred), `config/entity_registry/list`, `config/device_registry/list`, `config/area_registry/list` | – | setup wizard auto-detection (platform, device, area, category, translation key); verified, shapes in [Registries](#registries) | | `render_template` | `template` | last resort for integrations without actions | | `energy/get_prefs` | – | pre-fill from the Energy dashboard | @@ -454,8 +454,9 @@ PyPI wheels `forecast-solar 5.0.1`, `open-meteo-solar-forecast 0.1.32`. with `begin <= timestamp < end` for "current hour" (`forecast-solar 5.0.1` wheel, `forecast_solar/models.py` `_interval_value_sum`), i.e. it treats the key as the *start* of - the period. Entries with value 0 are dropped except non-midnight ones - (`val != 0 or time != 00:00:00`). `UNVERIFIED`: whether the vendor API itself + the period. Only an entry with value 0 that starts exactly at 00:00:00 is + dropped; every other zero is listed (kept: `val != 0 or time != 00:00:00`). + `UNVERIFIED`: whether the vendor API itself labels `watt_hours_period` keys as period start or end (api.forecast.solar unreachable); the HA library's own usage implies start. - `open_meteo_solar_forecast` (HACS, `energy.py`): `wh_period`, hourly, key = diff --git a/docs/user/compatibility.md b/docs/user/compatibility.md index 1725297..a8c49f4 100644 --- a/docs/user/compatibility.md +++ b/docs/user/compatibility.md @@ -36,6 +36,9 @@ a template only tested in the simulator has never run against real hardware. | Id | Kind | Name | Integration / platforms | Capabilities | Tested with | Unverified | |---|---|---|---|---|---|---| +| `forecast_solar.pv` | forecast | Forecast.Solar PV forecast (Energy dashboard) | any | `pv_curve` | not tested on hardware | 8 | +| `generic.pv_forecast` | forecast | Generic PV forecast (attribute of a template sensor) | `template` | `pv_curve` | not tested on hardware | 1 | +| `open_meteo_solar_forecast.pv` | forecast | Open-Meteo Solar Forecast | `open_meteo_solar_forecast` | `pv_curve` | not tested on hardware | 5 | | `solcast.pv` | forecast | Solcast PV forecast | `solcast_solar` | `pv_curve`, `pv_quantiles` | not tested on hardware | 4 | ## Regions diff --git a/frontend/e2e/wizard.spec.ts b/frontend/e2e/wizard.spec.ts index 92e650c..f36d1e5 100644 --- a/frontend/e2e/wizard.spec.ts +++ b/frontend/e2e/wizard.spec.ts @@ -136,7 +136,10 @@ test.describe('setup wizard', () => { import: { kind: 'fixed', fixed_price: 0.32 }, export: { kind: 'fixed', fixed_price: 0.08 }, }); - expect(await get('forecast')).toMatchObject({ sources: [{ template_id: 'solcast.pv' }] }); + // The demo keeps its other forecast sources (Open-Meteo, Forecast.Solar); Solcast is checked. + expect(await get('forecast')).toMatchObject({ + sources: expect.arrayContaining([expect.objectContaining({ template_id: 'solcast.pv' })]), + }); const devices = (await get('devices')).devices; expect(devices).toHaveLength(1); expect(devices[0]).toMatchObject({ template_id: 'generic.switch', priority: 0 }); diff --git a/frontend/src/lib/api/openapi.json b/frontend/src/lib/api/openapi.json index ae5e932..642e270 100644 --- a/frontend/src/lib/api/openapi.json +++ b/frontend/src/lib/api/openapi.json @@ -1,6 +1,93 @@ { "components": { "schemas": { + "AccuracyRecord": { + "additionalProperties": false, + "description": "Measured error of one source in one horizon bucket (WP5 bookkeeping).\n\n``mae_w`` is the mean absolute error and ``bias_w`` the mean signed error of P50 over\n``n`` intervals. **Sign convention (one for all of SunCore's forecast statistics):\nerror = actual - forecast**, so ``bias_w > 0`` means the source under-forecasts and\n``forecast + bias_w`` is the bias-corrected value; the risk model's net error\n(PV error - load error) uses the same convention.", + "properties": { + "bias_w": { + "title": "Bias W", + "type": "number" + }, + "horizon": { + "$ref": "#/components/schemas/HorizonBucket" + }, + "mae_w": { + "minimum": 0.0, + "title": "Mae W", + "type": "number" + }, + "n": { + "minimum": 0.0, + "title": "N", + "type": "integer" + }, + "source_id": { + "maxLength": 64, + "minLength": 1, + "pattern": "^[a-z0-9_.:-]+$", + "title": "Source Id", + "type": "string" + } + }, + "required": [ + "source_id", + "horizon", + "mae_w", + "bias_w", + "n" + ], + "title": "AccuracyRecord", + "type": "object" + }, + "AccuracySourceView": { + "additionalProperties": false, + "description": "Measured accuracy of one source; ``records`` only holds buckets with data.\n\nThe source id ``blended`` is the blended forecast the planner used.", + "properties": { + "computed_at": { + "anyOf": [ + { + "format": "date-time", + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Computed At" + }, + "records": { + "items": { + "$ref": "#/components/schemas/AccuracyRecord" + }, + "title": "Records", + "type": "array" + }, + "source_id": { + "title": "Source Id", + "type": "string" + }, + "window_days": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "title": "Window Days" + } + }, + "required": [ + "source_id", + "computed_at", + "window_days", + "records" + ], + "title": "AccuracySourceView", + "type": "object" + }, "ActionSeries": { "additionalProperties": false, "description": "Series returned by a read-only action (``return_response`` is implied).", @@ -664,6 +751,204 @@ "title": "BatteryTemplate", "type": "object" }, + "BlendWeightsView": { + "additionalProperties": false, + "description": "The weights of one horizon bucket (before the per-slot renormalisation over the sources\nthat have a value in a slot).", + "properties": { + "basis": { + "anyOf": [ + { + "enum": [ + "inverse_mae", + "equal_no_history" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Basis" + }, + "horizon": { + "$ref": "#/components/schemas/HorizonBucket" + }, + "manual": { + "title": "Manual", + "type": "boolean" + }, + "weights": { + "additionalProperties": { + "type": "number" + }, + "title": "Weights", + "type": "object" + } + }, + "required": [ + "horizon", + "basis", + "manual", + "weights" + ], + "title": "BlendWeightsView", + "type": "object" + }, + "BlendedView": { + "additionalProperties": false, + "description": "The forecast the planner uses: the sources on the 15-minute grid, bias-calibrated where\nthe calibration is active, blended by measured skill (P10/P50/P90 in W).\n\n``uncalibrated_p50_w`` is the blend of the sources as they said it; ``calibrated_p50_w`` the\nblend with every learnt calibration applied, active or not (``None`` while no source has\nlearnt anything), so the effect can be compared before it switches on.", + "properties": { + "calibrated_p50_w": { + "anyOf": [ + { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Calibrated P50 W" + }, + "calibration": { + "items": { + "$ref": "#/components/schemas/CalibrationView" + }, + "title": "Calibration", + "type": "array" + }, + "calibration_active": { + "title": "Calibration Active", + "type": "boolean" + }, + "excluded_sources": { + "items": { + "type": "string" + }, + "title": "Excluded Sources", + "type": "array" + }, + "method": { + "enum": [ + "weighted", + "single_source" + ], + "title": "Method", + "type": "string" + }, + "p10_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P10 W", + "type": "array" + }, + "p50_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P50 W", + "type": "array" + }, + "p90_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P90 W", + "type": "array" + }, + "sources": { + "items": { + "type": "string" + }, + "title": "Sources", + "type": "array" + }, + "stale": { + "title": "Stale", + "type": "boolean" + }, + "start": { + "format": "date-time", + "title": "Start", + "type": "string" + }, + "step_s": { + "exclusiveMinimum": 0.0, + "title": "Step S", + "type": "integer" + }, + "uncalibrated_p50_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "Uncalibrated P50 W", + "type": "array" + }, + "weights": { + "items": { + "$ref": "#/components/schemas/BlendWeightsView" + }, + "title": "Weights", + "type": "array" + } + }, + "required": [ + "method", + "sources", + "start", + "step_s", + "p10_w", + "p50_w", + "p90_w", + "uncalibrated_p50_w", + "calibrated_p50_w", + "calibration_active", + "stale", + "excluded_sources", + "weights", + "calibration" + ], + "title": "BlendedView", + "type": "object" + }, "CalibrationCharge": { "additionalProperties": false, "description": "Periodic full charge for the BMS's SoC calibration (off by default, question 11).\n\nWhen enabled the planner schedules a charge to ``target_soc_pct`` once every\n``interval_days`` into a sunny day (from PV; grid only if the policy allows it).", @@ -691,6 +976,76 @@ "title": "CalibrationCharge", "type": "object" }, + "CalibrationView": { + "additionalProperties": false, + "description": "The bias calibration of one source.\n\n``reason``: ``active`` (applied), ``collecting`` (learnt and stored but not applied before\n``min_history_days`` days with data), ``no_history`` (nothing learnt yet),\n``classifier_changed`` (learnt with the other weather classifier; waits).", + "properties": { + "active": { + "title": "Active", + "type": "boolean" + }, + "cells": { + "title": "Cells", + "type": "integer" + }, + "days_with_data": { + "title": "Days With Data", + "type": "integer" + }, + "factor_max": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Factor Max" + }, + "factor_min": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Factor Min" + }, + "min_history_days": { + "title": "Min History Days", + "type": "integer" + }, + "reason": { + "enum": [ + "active", + "collecting", + "no_history", + "classifier_changed" + ], + "title": "Reason", + "type": "string" + }, + "source_id": { + "title": "Source Id", + "type": "string" + } + }, + "required": [ + "source_id", + "active", + "reason", + "days_with_data", + "min_history_days", + "cells", + "factor_min", + "factor_max" + ], + "title": "CalibrationView", + "type": "object" + }, "Capability": { "description": "What a template provides.", "enum": [ @@ -1992,6 +2347,31 @@ "title": "EmergencyStopOut", "type": "object" }, + "EnergyDashboardSeries": { + "additionalProperties": false, + "description": "Series read from the WebSocket command ``energy/solar_forecast`` (Phase 4, forecasts only).\n\nThe command returns ``{config_entry_id: {\"wh_hours\": {iso_timestamp: Wh}}}`` for the forecast\nintegrations linked in the Energy dashboard (Forecast.Solar in Home Assistant core). The\ntemplate names no entity: the user binds the source to a config entry\n(``ForecastSourceConfig.config_entry_id``), or SunCore takes the only entry there is.\n``fields`` gives only ``value_unit`` (energy per period; no quantiles). The length of a\nperiod is inferred from the spacing of the timestamps; the template's ``resolution_min``\nis the expected length (used for a series of one point).", + "properties": { + "fields": { + "$ref": "#/components/schemas/SeriesFields" + }, + "missing_is_zero": { + "default": false, + "title": "Missing Is Zero", + "type": "boolean" + }, + "source": { + "const": "energy_solar_forecast", + "title": "Source", + "type": "string" + } + }, + "required": [ + "source", + "fields" + ], + "title": "EnergyDashboardSeries", + "type": "object" + }, "EnergyPrefsOut": { "additionalProperties": false, "description": "Energy dashboard import: suggestions only, never applied by the server.", @@ -2462,6 +2842,21 @@ "title": "FeedInTier", "type": "object" }, + "FetchStatus": { + "description": "What a fetch found (rendered as ``forecast.status.`` in the UI).", + "enum": [ + "ok", + "partial", + "no_data", + "not_configured", + "not_linked", + "ambiguous_entry", + "entry_not_found", + "error" + ], + "title": "FetchStatus", + "type": "string" + }, "FlowOut": { "additionalProperties": false, "description": "A value in W (or % for the state of charge) with its data quality.", @@ -2580,6 +2975,29 @@ "title": "ForceChargeSwitchControl", "type": "object" }, + "ForecastAccuracyView": { + "additionalProperties": false, + "description": "Accuracy per source and horizon bucket.\n\n``bias_w = actual - forecast`` (positive = the source under-forecasts); ``n`` counts\n15-minute daylight intervals (forecast or measurement above 0 W; clipped ones excluded).\nA record with ``n`` below ``min_samples`` is too thin to rely on.", + "properties": { + "min_samples": { + "title": "Min Samples", + "type": "integer" + }, + "sources": { + "items": { + "$ref": "#/components/schemas/AccuracySourceView" + }, + "title": "Sources", + "type": "array" + } + }, + "required": [ + "min_samples", + "sources" + ], + "title": "ForecastAccuracyView", + "type": "object" + }, "ForecastConfig": { "additionalProperties": false, "description": "PV forecast sources (schema v2: several sources, blending, learning settings).\n\nIndependent of the tariff so that the wizard can store the choice without prices.\nForecast uncertainty is handled by the learned risk model and the battery section's\n``risk`` level, not by a flat discount.", @@ -2612,6 +3030,18 @@ "additionalProperties": false, "description": "One PV forecast source: its template, the entities of its roles and its weight.\n\n``weight_mode = auto`` blends by measured skill (inverse MAE, WP5); ``manual`` uses\n``weight`` (relative, normalised over all sources by the blender).", "properties": { + "config_entry_id": { + "anyOf": [ + { + "pattern": "^[A-Za-z0-9_.:-]{1,64}$", + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Config Entry Id" + }, "entities": { "patternProperties": { "^[a-z][a-z0-9_]{0,31}$": { @@ -2667,6 +3097,16 @@ "title": "Capabilities", "type": "array" }, + "fallback_series": { + "anyOf": [ + { + "$ref": "#/components/schemas/ActionSeries" + }, + { + "type": "null" + } + ] + }, "forecast_kind": { "enum": [ "pv", @@ -2728,7 +3168,8 @@ "discriminator": { "mapping": { "action": "#/components/schemas/ActionSeries", - "attribute": "#/components/schemas/AttributeSeries" + "attribute": "#/components/schemas/AttributeSeries", + "energy_solar_forecast": "#/components/schemas/EnergyDashboardSeries" }, "propertyName": "source" }, @@ -2738,6 +3179,9 @@ }, { "$ref": "#/components/schemas/ActionSeries" + }, + { + "$ref": "#/components/schemas/EnergyDashboardSeries" } ], "title": "Series" @@ -3086,6 +3530,17 @@ "title": "HistoryRestoreOut", "type": "object" }, + "HorizonBucket": { + "description": "Lead-time bucket the accuracy of a forecast is tracked in.", + "enum": [ + "0_6h", + "6_24h", + "24_48h", + "48h_plus" + ], + "title": "HorizonBucket", + "type": "string" + }, "ImportResultOut": { "additionalProperties": false, "properties": { @@ -4143,6 +4598,177 @@ "title": "PvArray", "type": "object" }, + "PvForecastView": { + "additionalProperties": false, + "properties": { + "blended": { + "anyOf": [ + { + "$ref": "#/components/schemas/BlendedView" + }, + { + "type": "null" + } + ] + }, + "generated_at": { + "format": "date-time", + "title": "Generated At", + "type": "string" + }, + "hours": { + "title": "Hours", + "type": "integer" + }, + "sources": { + "items": { + "$ref": "#/components/schemas/PvSourceView" + }, + "title": "Sources", + "type": "array" + } + }, + "required": [ + "generated_at", + "hours", + "sources", + "blended" + ], + "title": "PvForecastView", + "type": "object" + }, + "PvSeriesView": { + "additionalProperties": false, + "description": "P10/P50/P90 in W on one regular grid: ``p50_w[i]`` covers ``[start + i*step_s, ...)``.", + "properties": { + "p10_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P10 W", + "type": "array" + }, + "p50_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P50 W", + "type": "array" + }, + "p90_w": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "P90 W", + "type": "array" + }, + "start": { + "format": "date-time", + "title": "Start", + "type": "string" + }, + "step_s": { + "exclusiveMinimum": 0.0, + "title": "Step S", + "type": "integer" + } + }, + "required": [ + "start", + "step_s", + "p10_w", + "p50_w", + "p90_w" + ], + "title": "PvSeriesView", + "type": "object" + }, + "PvSourceView": { + "additionalProperties": false, + "description": "One source: its latest good series and how fresh it is.", + "properties": { + "age_s": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Age S" + }, + "fetched_at": { + "anyOf": [ + { + "format": "date-time", + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Fetched At" + }, + "filled_slots": { + "title": "Filled Slots", + "type": "integer" + }, + "series": { + "anyOf": [ + { + "$ref": "#/components/schemas/PvSeriesView" + }, + { + "type": "null" + } + ] + }, + "source_id": { + "title": "Source Id", + "type": "string" + }, + "stale": { + "title": "Stale", + "type": "boolean" + }, + "status": { + "$ref": "#/components/schemas/FetchStatus" + } + }, + "required": [ + "source_id", + "status", + "fetched_at", + "age_s", + "stale", + "filled_slots", + "series" + ], + "title": "PvSourceView", + "type": "object" + }, "Quality": { "description": "Data quality, ordered best to worst: ok < stale < unavailable < invalid.", "enum": [ @@ -6856,6 +7482,68 @@ "summary": "Entities" } }, + "/api/v1/forecast/accuracy": { + "get": { + "description": "MAE and bias per source and horizon bucket (0-6 h, 6-24 h, 24-48 h) over the last 14\ndays, plus the source id ``blended`` (the blended forecast); ``bias_w = actual -\nforecast`` (positive: the source under-forecasts); ``n`` counts 15-minute intervals;\nclipped quarter hours are excluded; computed nightly.", + "operationId": "forecast_accuracy_api_v1_forecast_accuracy_get", + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ForecastAccuracyView" + } + } + }, + "description": "Successful Response" + } + }, + "summary": "Forecast Accuracy" + } + }, + "/api/v1/forecast/pv": { + "get": { + "description": "Per source the latest good P10/P50/P90 in W from the current quarter hour on, with\nits freshness, and ``blended``: the forecast the planner uses (sources bias-calibrated\nwhere active, blended by inverse 14-day MAE), its weights per horizon bucket, the\ncalibration state per source, and the blend without and with every learnt\ncalibration for comparison.", + "operationId": "forecast_pv_api_v1_forecast_pv_get", + "parameters": [ + { + "in": "query", + "name": "hours", + "required": false, + "schema": { + "default": 48, + "maximum": 96, + "minimum": 1, + "title": "Hours", + "type": "integer" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PvForecastView" + } + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ValidationErrorOut" + } + } + }, + "description": "Validation error" + } + }, + "summary": "Forecast Pv" + } + }, "/api/v1/info": { "get": { "operationId": "info_api_v1_info_get", diff --git a/frontend/src/lib/api/schema.d.ts b/frontend/src/lib/api/schema.d.ts index e69c60c..414ab59 100644 --- a/frontend/src/lib/api/schema.d.ts +++ b/frontend/src/lib/api/schema.d.ts @@ -369,6 +369,53 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/forecast/accuracy": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Forecast Accuracy + * @description MAE and bias per source and horizon bucket (0-6 h, 6-24 h, 24-48 h) over the last 14 + * days, plus the source id ``blended`` (the blended forecast); ``bias_w = actual - + * forecast`` (positive: the source under-forecasts); ``n`` counts 15-minute intervals; + * clipped quarter hours are excluded; computed nightly. + */ + get: operations["forecast_accuracy_api_v1_forecast_accuracy_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v1/forecast/pv": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** + * Forecast Pv + * @description Per source the latest good P10/P50/P90 in W from the current quarter hour on, with + * its freshness, and ``blended``: the forecast the planner uses (sources bias-calibrated + * where active, blended by inverse 14-day MAE), its weights per horizon bucket, the + * calibration state per source, and the blend without and with every learnt + * calibration for comparison. + */ + get: operations["forecast_pv_api_v1_forecast_pv_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/info": { parameters: { query?: never; @@ -604,6 +651,43 @@ export interface paths { export type webhooks = Record; export interface components { schemas: { + /** + * AccuracyRecord + * @description Measured error of one source in one horizon bucket (WP5 bookkeeping). + * + * ``mae_w`` is the mean absolute error and ``bias_w`` the mean signed error of P50 over + * ``n`` intervals. **Sign convention (one for all of SunCore's forecast statistics): + * error = actual - forecast**, so ``bias_w > 0`` means the source under-forecasts and + * ``forecast + bias_w`` is the bias-corrected value; the risk model's net error + * (PV error - load error) uses the same convention. + */ + AccuracyRecord: { + /** Bias W */ + bias_w: number; + horizon: components["schemas"]["HorizonBucket"]; + /** Mae W */ + mae_w: number; + /** N */ + n: number; + /** Source Id */ + source_id: string; + }; + /** + * AccuracySourceView + * @description Measured accuracy of one source; ``records`` only holds buckets with data. + * + * The source id ``blended`` is the blended forecast the planner used. + */ + AccuracySourceView: { + /** Computed At */ + computed_at: string | null; + /** Records */ + records: components["schemas"]["AccuracyRecord"][]; + /** Source Id */ + source_id: string; + /** Window Days */ + window_days: number | null; + }; /** * ActionSeries * @description Series returned by a read-only action (``return_response`` is implied). @@ -899,6 +983,67 @@ export interface components { /** Unverified */ unverified?: string[]; }; + /** + * BlendWeightsView + * @description The weights of one horizon bucket (before the per-slot renormalisation over the sources + * that have a value in a slot). + */ + BlendWeightsView: { + /** Basis */ + basis: ("inverse_mae" | "equal_no_history") | null; + horizon: components["schemas"]["HorizonBucket"]; + /** Manual */ + manual: boolean; + /** Weights */ + weights: { + [key: string]: number; + }; + }; + /** + * BlendedView + * @description The forecast the planner uses: the sources on the 15-minute grid, bias-calibrated where + * the calibration is active, blended by measured skill (P10/P50/P90 in W). + * + * ``uncalibrated_p50_w`` is the blend of the sources as they said it; ``calibrated_p50_w`` the + * blend with every learnt calibration applied, active or not (``None`` while no source has + * learnt anything), so the effect can be compared before it switches on. + */ + BlendedView: { + /** Calibrated P50 W */ + calibrated_p50_w: (number | null)[] | null; + /** Calibration */ + calibration: components["schemas"]["CalibrationView"][]; + /** Calibration Active */ + calibration_active: boolean; + /** Excluded Sources */ + excluded_sources: string[]; + /** + * Method + * @enum {string} + */ + method: "weighted" | "single_source"; + /** P10 W */ + p10_w: (number | null)[]; + /** P50 W */ + p50_w: (number | null)[]; + /** P90 W */ + p90_w: (number | null)[]; + /** Sources */ + sources: string[]; + /** Stale */ + stale: boolean; + /** + * Start + * Format: date-time + */ + start: string; + /** Step S */ + step_s: number; + /** Uncalibrated P50 W */ + uncalibrated_p50_w: (number | null)[]; + /** Weights */ + weights: components["schemas"]["BlendWeightsView"][]; + }; /** * CalibrationCharge * @description Periodic full charge for the BMS's SoC calibration (off by default, question 11). @@ -923,6 +1068,35 @@ export interface components { */ target_soc_pct: number; }; + /** + * CalibrationView + * @description The bias calibration of one source. + * + * ``reason``: ``active`` (applied), ``collecting`` (learnt and stored but not applied before + * ``min_history_days`` days with data), ``no_history`` (nothing learnt yet), + * ``classifier_changed`` (learnt with the other weather classifier; waits). + */ + CalibrationView: { + /** Active */ + active: boolean; + /** Cells */ + cells: number; + /** Days With Data */ + days_with_data: number; + /** Factor Max */ + factor_max: number | null; + /** Factor Min */ + factor_min: number | null; + /** Min History Days */ + min_history_days: number; + /** + * Reason + * @enum {string} + */ + reason: "active" | "collecting" | "no_history" | "classifier_changed"; + /** Source Id */ + source_id: string; + }; /** * Capability * @description What a template provides. @@ -1342,6 +1516,31 @@ export interface components { /** Active */ active: boolean; }; + /** + * EnergyDashboardSeries + * @description Series read from the WebSocket command ``energy/solar_forecast`` (Phase 4, forecasts only). + * + * The command returns ``{config_entry_id: {"wh_hours": {iso_timestamp: Wh}}}`` for the forecast + * integrations linked in the Energy dashboard (Forecast.Solar in Home Assistant core). The + * template names no entity: the user binds the source to a config entry + * (``ForecastSourceConfig.config_entry_id``), or SunCore takes the only entry there is. + * ``fields`` gives only ``value_unit`` (energy per period; no quantiles). The length of a + * period is inferred from the spacing of the timestamps; the template's ``resolution_min`` + * is the expected length (used for a series of one point). + */ + EnergyDashboardSeries: { + fields: components["schemas"]["SeriesFields"]; + /** + * Missing Is Zero + * @default false + */ + missing_is_zero: boolean; + /** + * @description discriminator enum property added by openapi-typescript + * @enum {string} + */ + source: "energy_solar_forecast"; + }; /** * EnergyPrefsOut * @description Energy dashboard import: suggestions only, never applied by the server. @@ -1501,6 +1700,12 @@ export interface components { /** Valid Until */ valid_until?: string | null; }; + /** + * FetchStatus + * @description What a fetch found (rendered as ``forecast.status.`` in the UI). + * @enum {string} + */ + FetchStatus: "ok" | "partial" | "no_data" | "not_configured" | "not_linked" | "ambiguous_entry" | "entry_not_found" | "error"; /** * FlowOut * @description A value in W (or % for the state of charge) with its data quality. @@ -1547,6 +1752,20 @@ export interface components { */ kind: "force_charge"; }; + /** + * ForecastAccuracyView + * @description Accuracy per source and horizon bucket. + * + * ``bias_w = actual - forecast`` (positive = the source under-forecasts); ``n`` counts + * 15-minute daylight intervals (forecast or measurement above 0 W; clipped ones excluded). + * A record with ``n`` below ``min_samples`` is too thin to rely on. + */ + ForecastAccuracyView: { + /** Min Samples */ + min_samples: number; + /** Sources */ + sources: components["schemas"]["AccuracySourceView"][]; + }; /** * ForecastConfig * @description PV forecast sources (schema v2: several sources, blending, learning settings). @@ -1579,6 +1798,8 @@ export interface components { * ``weight`` (relative, normalised over all sources by the blender). */ ForecastSourceConfig: { + /** Config Entry Id */ + config_entry_id?: string | null; /** Entities */ entities?: { [key: string]: components["schemas"]["EntityRef"]; @@ -1598,6 +1819,7 @@ export interface components { ForecastTemplate: { /** Capabilities */ capabilities: components["schemas"]["Capability"][]; + fallback_series?: components["schemas"]["ActionSeries"] | null; /** * Forecast Kind * @enum {string} @@ -1626,7 +1848,7 @@ export interface components { */ schema_version: 1; /** Series */ - series: components["schemas"]["AttributeSeries"] | components["schemas"]["ActionSeries"]; + series: components["schemas"]["AttributeSeries"] | components["schemas"]["ActionSeries"] | components["schemas"]["EnergyDashboardSeries"]; /** Source */ source: components["schemas"]["SourceRef"][]; /** Tested With */ @@ -1754,6 +1976,12 @@ export interface components { /** Section */ section: string; }; + /** + * HorizonBucket + * @description Lead-time bucket the accuracy of a forecast is tracked in. + * @enum {string} + */ + HorizonBucket: "0_6h" | "6_24h" | "24_48h" | "48h_plus"; /** ImportResultOut */ ImportResultOut: { /** Imported Sections */ @@ -2196,6 +2424,56 @@ export interface components { /** Tilt Deg */ tilt_deg?: number | null; }; + /** PvForecastView */ + PvForecastView: { + blended: components["schemas"]["BlendedView"] | null; + /** + * Generated At + * Format: date-time + */ + generated_at: string; + /** Hours */ + hours: number; + /** Sources */ + sources: components["schemas"]["PvSourceView"][]; + }; + /** + * PvSeriesView + * @description P10/P50/P90 in W on one regular grid: ``p50_w[i]`` covers ``[start + i*step_s, ...)``. + */ + PvSeriesView: { + /** P10 W */ + p10_w: (number | null)[]; + /** P50 W */ + p50_w: (number | null)[]; + /** P90 W */ + p90_w: (number | null)[]; + /** + * Start + * Format: date-time + */ + start: string; + /** Step S */ + step_s: number; + }; + /** + * PvSourceView + * @description One source: its latest good series and how fresh it is. + */ + PvSourceView: { + /** Age S */ + age_s: number | null; + /** Fetched At */ + fetched_at: string | null; + /** Filled Slots */ + filled_slots: number; + series: components["schemas"]["PvSeriesView"] | null; + /** Source Id */ + source_id: string; + /** Stale */ + stale: boolean; + status: components["schemas"]["FetchStatus"]; + }; /** * Quality * @description Data quality, ordered best to worst: ok < stale < unavailable < invalid. @@ -3746,6 +4024,57 @@ export interface operations { }; }; }; + forecast_accuracy_api_v1_forecast_accuracy_get: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["ForecastAccuracyView"]; + }; + }; + }; + }; + forecast_pv_api_v1_forecast_pv_get: { + parameters: { + query?: { + hours?: number; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["PvForecastView"]; + }; + }; + /** @description Validation error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["ValidationErrorOut"]; + }; + }; + }; + }; info_api_v1_info_get: { parameters: { query?: never; diff --git a/frontend/src/lib/i18n/locales/de.json b/frontend/src/lib/i18n/locales/de.json index 0a28c0f..60d6159 100644 --- a/frontend/src/lib/i18n/locales/de.json +++ b/frontend/src/lib/i18n/locales/de.json @@ -756,6 +756,16 @@ "template.solcast.pv.name": "Solcast-PV-Prognose", "template.solcast.pv.role.forecast_today": "Solcast-Sensor für die Prognose von heute (kWh)", "template.solcast.pv.role.forecast_tomorrow": "Solcast-Sensor für die Prognose von morgen (kWh)", + "template.solcast.pv.role.forecast_d3": "Solcast-Sensor für die Prognose von Tag 3 (kWh, optional)", + "template.solcast.pv.role.forecast_d4": "Solcast-Sensor für die Prognose von Tag 4 (kWh, optional)", + "template.forecast_solar.pv.name": "Forecast.Solar-PV-Prognose (Energie-Dashboard)", + "template.open_meteo_solar_forecast.pv.name": "Open-Meteo Solar Forecast", + "template.open_meteo_solar_forecast.pv.role.energy_today": "Open-Meteo-Solar-Forecast-Sensor für heute (Energieerzeugung heute)", + "template.open_meteo_solar_forecast.pv.role.energy_tomorrow": "Open-Meteo-Solar-Forecast-Sensor für morgen (Energieerzeugung morgen)", + "template.open_meteo_solar_forecast.pv.role.energy_d2": "Open-Meteo-Solar-Forecast-Sensor für Tag 2 (optional)", + "template.open_meteo_solar_forecast.pv.role.energy_d3": "Open-Meteo-Solar-Forecast-Sensor für Tag 3 (optional)", + "template.generic.pv_forecast.name": "Allgemeine PV-Prognose (Attribut eines Template-Sensors)", + "template.generic.pv_forecast.role.forecast": "Sensor, dessen Attribut „forecast“ Punkte mit „start“ und „power_w“ auflistet", "template.tibber.price.name": "Tibber dynamischer Preis", "template.tibber.price.role.price": "Aktueller Strompreis-Sensor von Tibber (Währung pro kWh)", "template.untested": "Nicht an echter Hardware geprüft", diff --git a/frontend/src/lib/i18n/locales/en.json b/frontend/src/lib/i18n/locales/en.json index 0bf9c58..e6e777e 100644 --- a/frontend/src/lib/i18n/locales/en.json +++ b/frontend/src/lib/i18n/locales/en.json @@ -756,6 +756,16 @@ "template.solcast.pv.name": "Solcast PV forecast", "template.solcast.pv.role.forecast_today": "Solcast sensor for today's forecast (kWh)", "template.solcast.pv.role.forecast_tomorrow": "Solcast sensor for tomorrow's forecast (kWh)", + "template.solcast.pv.role.forecast_d3": "Solcast sensor for the forecast of day 3 (kWh, optional)", + "template.solcast.pv.role.forecast_d4": "Solcast sensor for the forecast of day 4 (kWh, optional)", + "template.forecast_solar.pv.name": "Forecast.Solar PV forecast (Energy dashboard)", + "template.open_meteo_solar_forecast.pv.name": "Open-Meteo Solar Forecast", + "template.open_meteo_solar_forecast.pv.role.energy_today": "Open-Meteo Solar Forecast sensor for today (energy production today)", + "template.open_meteo_solar_forecast.pv.role.energy_tomorrow": "Open-Meteo Solar Forecast sensor for tomorrow (energy production tomorrow)", + "template.open_meteo_solar_forecast.pv.role.energy_d2": "Open-Meteo Solar Forecast sensor for day 2 (optional)", + "template.open_meteo_solar_forecast.pv.role.energy_d3": "Open-Meteo Solar Forecast sensor for day 3 (optional)", + "template.generic.pv_forecast.name": "Generic PV forecast (attribute of a template sensor)", + "template.generic.pv_forecast.role.forecast": "Sensor whose attribute “forecast” lists points with “start” and “power_w”", "template.tibber.price.name": "Tibber dynamic price", "template.tibber.price.role.price": "Current electricity price sensor of Tibber (currency per kWh)", "template.untested": "Not verified with real hardware", diff --git a/suncore/api/app.py b/suncore/api/app.py index 41f46ca..9e6e152 100644 --- a/suncore/api/app.py +++ b/suncore/api/app.py @@ -86,6 +86,7 @@ from suncore.core.decisions import Decision, DecisionKind from suncore.core.site import SiteConfig from suncore.core.wizard import PreviewTooLargeError +from suncore.forecast.views import MAX_VIEW_HOURS, ForecastAccuracyView, PvForecastView from suncore.storage.errors import ( ConfigCorruptError, HistoryEntryNotFoundError, @@ -170,6 +171,7 @@ async def info() -> InfoOut: _register_sections(app, services) _register_history(app, services) _register_control(app, services) + _register_forecast(app, services) @app.get( f"{API_PREFIX}/entities", @@ -268,6 +270,29 @@ async def decisions( return await services.decisions(limit, device_id=device_id, kinds=kind) +def _register_forecast(app: FastAPI, services: Services) -> None: + """PV forecast and its measured accuracy (Phase 4 WP5; read only, no UI yet).""" + + @app.get(f"{API_PREFIX}/forecast/pv", response_model=PvForecastView) + async def forecast_pv( + hours: int = Query(default=48, ge=1, le=MAX_VIEW_HOURS), + ) -> PvForecastView: + """Per source the latest good P10/P50/P90 in W from the current quarter hour on, with + its freshness, and ``blended``: the forecast the planner uses (sources bias-calibrated + where active, blended by inverse 14-day MAE), its weights per horizon bucket, the + calibration state per source, and the blend without and with every learnt + calibration for comparison.""" + return await services.pv_forecast(hours) + + @app.get(f"{API_PREFIX}/forecast/accuracy", response_model=ForecastAccuracyView) + async def forecast_accuracy() -> ForecastAccuracyView: + """MAE and bias per source and horizon bucket (0-6 h, 6-24 h, 24-48 h) over the last 14 + days, plus the source id ``blended`` (the blended forecast); ``bias_w = actual - + forecast`` (positive: the source under-forecasts); ``n`` counts 15-minute intervals; + clipped quarter hours are excluded; computed nightly.""" + return await services.forecast_accuracy() + + def _register_control(app: FastAPI, services: Services) -> None: """Control state, emergency stop, overrides, proposals and operating modes (WP9).""" diff --git a/suncore/api/services.py b/suncore/api/services.py index 216380c..12ea9cb 100644 --- a/suncore/api/services.py +++ b/suncore/api/services.py @@ -22,6 +22,7 @@ from suncore.core.detection import DetectedEntity, EnergySuggestions from suncore.core.site import SiteConfig from suncore.core.wizard import CheckItem, PreviewItem +from suncore.forecast.views import ForecastAccuracyView, PvForecastView from suncore.storage import HistoryEntry from suncore.templates import TemplateCatalog @@ -164,6 +165,18 @@ async def decisions( """Newest first; ``device_id`` filters by the decision's device, ``kinds`` by kind.""" ... + # --- forecasts (Phase 4 WP5) --------------------------------------------------------- + + async def pv_forecast(self, hours: int) -> PvForecastView: + """The PV forecast of every configured source for the next ``hours`` (W, P10/P50/P90), + its freshness, and the blended forecast with its weights and calibration state.""" + ... + + async def forecast_accuracy(self) -> ForecastAccuracyView: + """MAE and bias (actual - forecast) per source and horizon bucket from the nightly job, + including the blended forecast (source id ``blended``).""" + ... + # --- control (Phase 3 WP9) --------------------------------------------------------- def control_state(self) -> ControlState: diff --git a/suncore/app/adapters.py b/suncore/app/adapters.py index 1c7e2d5..df34ff4 100644 --- a/suncore/app/adapters.py +++ b/suncore/app/adapters.py @@ -1,14 +1,90 @@ # SPDX-License-Identifier: Apache-2.0 """Composition of the tariff and forecast adapters (Phase 4 WP4/WP5). -This module will build the price and forecast sources from the configured templates and hand -them the Home Assistant connection. It is the only module of ``suncore.app`` that may call +This module builds the price and forecast sources from the configured templates and hands them +the Home Assistant connection. It is the only module of ``suncore.app`` that may call ``HAConnection.call_action_response`` (``tests/safety/test_boundary.py``, rule 7): the call is a read-only action with ``return_response`` and is rate limited per ``(domain, service)`` by the connection (``ActionRateLimiter``). An adapter passes the polling interval of its template as ``min_interval_s`` and treats ``ActionRateLimitedError`` as "use the last value". -Empty on purpose until the first adapter lands (PR 4-C). +The PV forecast sources read through :class:`HaPvDataReader`: attributes of the entity cache (no +extra Home Assistant call), the Energy dashboard's ``energy/solar_forecast`` through +``suncore.ha.fetch_solar_forecast`` (rule 9 keeps the command inside ``suncore.ha``) and the +read-only action ``solcast_solar.query_forecast_data``. """ from __future__ import annotations + +from collections.abc import Awaitable, Callable, Mapping +from datetime import datetime +from typing import Any + +from suncore.core.config import ForecastConfig +from suncore.forecast.pv.factory import build_sources +from suncore.forecast.pv.service import ( + BlendContext, + ForecastStore, + PvForecastService, + SourceSettings, +) +from suncore.ha import EntityCache, HAConnection, fetch_solar_forecast +from suncore.ha.cache import UNAVAILABLE_STATES +from suncore.templates import TemplateCatalog + + +class HaPvDataReader: + """The :class:`~suncore.forecast.pv.adapter.PvDataReader` backed by Home Assistant.""" + + def __init__(self, connection: HAConnection, cache: EntityCache) -> None: + self._connection = connection + self._cache = cache + + def attributes(self, entity_id: str) -> Mapping[str, Any] | None: + entry = self._cache.get(entity_id) + if entry is None or entry.state.state in UNAVAILABLE_STATES: + return None + return entry.state.attributes + + async def solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + return await fetch_solar_forecast(self._connection) + + async def action_response( + self, domain: str, service: str, data: Mapping[str, Any], *, min_interval_s: float + ) -> Mapping[str, Any]: + return await self._connection.call_action_response( + domain, service, data, min_interval_s=min_interval_s + ) + + +def build_pv_forecast_service( + config: ForecastConfig | None, + *, + catalog: TemplateCatalog, + connection: HAConnection, + cache: EntityCache, + clock: Callable[[], datetime], + sleep: Callable[[float], Awaitable[None]], + store: ForecastStore | None, + context: Callable[[], BlendContext] = BlendContext, +) -> PvForecastService | None: + """The refresh service of the configured PV sources; ``None`` if none is usable.""" + sources = build_sources(config, catalog, HaPvDataReader(connection, cache)) + if not sources or config is None: + return None + settings = { + item.template_id: SourceSettings(weight_mode=item.weight_mode, manual_weight=item.weight) + for item in config.sources + } + return PvForecastService( + sources, + clock=clock, + sleep=sleep, + store=store, + settings=settings, + min_history_days=config.learning.min_history_days, + context=context, + ) + + +__all__ = ["HaPvDataReader", "build_pv_forecast_service"] diff --git a/suncore/app/openapi_stub.py b/suncore/app/openapi_stub.py index 5d8ca79..7345f11 100644 --- a/suncore/app/openapi_stub.py +++ b/suncore/app/openapi_stub.py @@ -4,6 +4,7 @@ from __future__ import annotations from collections.abc import Callable, Collection, Mapping, Sequence +from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -18,6 +19,8 @@ from suncore.core.detection import DetectedEntity, EnergySuggestions from suncore.core.site import SiteConfig from suncore.core.wizard import CheckItem, PreviewItem +from suncore.forecast.accuracy import accuracy_view +from suncore.forecast.views import ForecastAccuracyView, PvForecastView from suncore.storage import HistoryEntry from suncore.templates import TemplateCatalog @@ -77,6 +80,12 @@ async def decisions( ) -> list[Decision]: return [] + async def pv_forecast(self, hours: int) -> PvForecastView: + return PvForecastView(generated_at=datetime.now(UTC), hours=hours, sources=[], blended=None) + + async def forecast_accuracy(self) -> ForecastAccuracyView: + return accuracy_view([], {}) + def control_state(self) -> ControlState: return ControlState( ts=None, diff --git a/suncore/app/runtime.py b/suncore/app/runtime.py index 8b82efb..b03dde2 100644 --- a/suncore/app/runtime.py +++ b/suncore/app/runtime.py @@ -26,6 +26,7 @@ UnknownRoleError, UnknownTemplateError, ) +from suncore.app.adapters import build_pv_forecast_service from suncore.app.cadence import device_cadences, merge_cadences from suncore.app.control import ControlStack from suncore.app.history import rebuild_history @@ -33,7 +34,13 @@ from suncore.app.recorder import FLUSH_INTERVAL_S, MeasurementRecorder from suncore.app.settings import SUPERVISOR_INGRESS_PEER, Settings from suncore.core.balance import FlowSnapshot, compute_flows -from suncore.core.config import DevicesConfig, GoalsConfig, OperationConfig, RegionConfig +from suncore.core.config import ( + DevicesConfig, + ForecastConfig, + GoalsConfig, + OperationConfig, + RegionConfig, +) from suncore.core.control import ControlSettings from suncore.core.controlstate import ( ControlState, @@ -52,6 +59,13 @@ from suncore.core.signals import Sample from suncore.core.site import SiteConfig from suncore.core.wizard import CheckItem, PreviewItem, collect_refs, preview, run_system_check +from suncore.forecast.accuracy import accuracy_view, load_accuracy +from suncore.forecast.pv.calibration import load_bias +from suncore.forecast.pv.clearsky import plant_geometry +from suncore.forecast.pv.clipping import export_limit_w +from suncore.forecast.pv.learning import BLENDED_SOURCE_ID, LearningContext, run_nightly +from suncore.forecast.pv.service import BlendContext, PvForecastService +from suncore.forecast.views import ForecastAccuracyView, PvForecastView from suncore.ha import ( SITE_ROLES, CommandError, @@ -128,6 +142,9 @@ class Runtime: devices: DevicesConfig | None = None region: RegionConfig | None = None goals: GoalsConfig | None = None + forecast: ForecastConfig | None = None + pv_service: PvForecastService | None = None + """Refreshes the configured PV forecast sources (``None`` = none configured).""" timeseries_db: Database | None = None """``timeseries.db``; ``None`` = unavailable (too new, unopenable): control is unaffected.""" timeseries: TimeseriesStore | None = None @@ -160,6 +177,8 @@ class Runtime: _flush_task: asyncio.Task[None] | None = None _maintenance_task: asyncio.Task[None] | None = None _history_task: asyncio.Task[None] | None = None + _pv_task: asyncio.Task[None] | None = None + _accuracy_task: asyncio.Future[None] | None = None _history_done: bool = False """The statistics rebuild ran (or was not needed) in this process.""" _stack_lock: asyncio.Lock = field(default_factory=asyncio.Lock) @@ -169,6 +188,8 @@ class Runtime: _tz_learnt: bool = False """Home Assistant's time zone was asked for (or a region makes it irrelevant).""" _tz_lock: asyncio.Lock = field(default_factory=asyncio.Lock) + _ha_location: tuple[float, float] | None = None + """Home Assistant's ``latitude``/``longitude`` (``get_config``), for the PV clear-sky model.""" @property def gate(self) -> SafetyGate: @@ -187,6 +208,8 @@ async def prepare(self) -> None: """ await self._load_live_sections() self.stack = self._new_stack() + self.pv_service = self._new_pv_service() + await self._load_pv_learning() async def start(self) -> None: """Wire the control stack and the connection and start connecting in the background. @@ -208,6 +231,7 @@ async def start(self) -> None: self.stack.start_ticking() self._connect_task = asyncio.create_task(self._connect(), name="suncore-ha-connect") self._prune_task = asyncio.create_task(self._prune_loop(), name="suncore-prune") + self._start_pv_task() if self.timeseries is not None: self._flush_task = asyncio.create_task(self._flush_loop(), name="suncore-ts-flush") self._maintenance_task = asyncio.create_task( @@ -225,6 +249,9 @@ async def _load_live_sections(self) -> None: goals = await self.config.get("goals") assert goals is None or isinstance(goals, GoalsConfig) self.goals = goals + forecast = await self.config.get("forecast") + assert forecast is None or isinstance(forecast, ForecastConfig) + self.forecast = forecast await self._load_modes() async def _load_modes(self) -> None: @@ -363,8 +390,129 @@ async def maintain_timeseries(self) -> bool: except Exception: log.exception("timeseries.maintenance_failed") return False + self._schedule_forecast_accuracy() return True + def _schedule_forecast_accuracy(self) -> None: + """Run the accuracy statistics in the background (nightly timing unchanged).""" + if ( + not self._started + or self.pv_service is None + or (self._accuracy_task is not None and not self._accuracy_task.done()) + ): + return + self._accuracy_task = asyncio.ensure_future(self.update_forecast_accuracy()) + + async def update_forecast_accuracy(self) -> None: + """The nightly PV forecast learning: clipping, accuracy of the sources and of the + blend, bias tables (never raises); the results go straight to the service.""" + if self.timeseries is None or self.learned is None or self.pv_service is None: + return + if self._ha_location is None: + await self._learn_ha_location() + service = self.pv_service + context = self._blend_context() + try: + await run_nightly( + self.timeseries, + self.learned, + service.source_ids, + self.clock(), + LearningContext( + tz=context.tz, plant=context.plant, export_limit_w=export_limit_w(self.site) + ), + ) + except Exception: + log.exception("forecast.accuracy_failed") + return + if self.pv_service is service: + await self._load_pv_learning() + + # --- PV forecast (Phase 4 WP5) --------------------------------------------------- + + def _new_pv_service(self) -> PvForecastService | None: + return build_pv_forecast_service( + self.forecast, + catalog=self.template_catalog, + connection=self.connection, + cache=self.cache, + clock=self.clock, + sleep=self.control_sleep, + store=self.timeseries, + context=self._blend_context, + ) + + def _blend_context(self) -> BlendContext: + """Time zone and plant geometry for the blend (read anew on every use: the site, the + region and Home Assistant's location may change while the service runs).""" + name = self.region.timezone if self.region is not None else self._ha_timezone + tz: tzinfo = UTC + if name is not None: + with contextlib.suppress(ZoneInfoNotFoundError, ValueError, OSError): + tz = ZoneInfo(name) + lat, lon = self._ha_location or (None, None) + arrays = [] if self.site is None else self.site.pv + return BlendContext(tz=tz, plant=plant_geometry(lat, lon, arrays)) + + async def _learn_ha_location(self) -> None: + """Home Assistant's ``latitude``/``longitude`` from ``get_config`` (verified in + ``homeassistant/core_config.py`` ``Config.as_dict``); a failure keeps the fallback.""" + try: + config = await self.connection.get_config() + except (HAError, TimeoutError) as exc: + log.warning("ha.get_config_failed", error=str(exc)) + return + lat, lon = config.get("latitude"), config.get("longitude") + if ( + isinstance(lat, int | float) + and isinstance(lon, int | float) + and not isinstance(lat, bool) + and not isinstance(lon, bool) + ): + self._ha_location = (float(lat), float(lon)) + + async def _load_pv_learning(self) -> None: + """Hand the stored accuracy records and bias tables to the PV service.""" + service = self.pv_service + if service is None or self.learned is None: + return + try: + accuracy = await load_accuracy(self.learned, service.source_ids) + bias = await load_bias(self.learned) + except Exception: + log.exception("forecast.learning_load_failed") + return + service.set_learning( + {k: v.records for k, v in accuracy.items()}, + {k: v for k, v in bias.items() if k in service.source_ids}, + ) + + def _start_pv_task(self) -> None: + if self.pv_service is not None and (self._pv_task is None or self._pv_task.done()): + self._pv_task = asyncio.create_task(self.pv_service.run(), name="suncore-pv-forecast") + + async def _rebuild_pv_service(self) -> None: + """Replace the PV forecast service after a ``forecast`` section change.""" + task, self._pv_task = self._pv_task, None + if task is not None: + task.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await task + self.pv_service = self._new_pv_service() + await self._load_pv_learning() + if self._started: + self._start_pv_task() + + async def pv_forecast(self, hours: int) -> PvForecastView: + if self.pv_service is None: + return PvForecastView(generated_at=self.clock(), hours=hours, sources=[], blended=None) + return self.pv_service.view(hours) + + async def forecast_accuracy(self) -> ForecastAccuracyView: + ids = [] if self.pv_service is None else [*self.pv_service.source_ids, BLENDED_SOURCE_ID] + stored = {} if self.learned is None else await load_accuracy(self.learned, ids) + return accuracy_view(ids, stored) + def next_maintenance_after(self, moment: datetime) -> datetime: """The first ``maintenance_time`` (site time zone, DST safe) strictly after ``moment``.""" tz = self._timezone() @@ -454,6 +602,7 @@ async def _connect(self) -> None: log.error("ha.connect_failed", error_type=type(exc).__name__, error=str(exc)) return await self._prepare_control() + await self._learn_ha_location() self._start_history_rebuild() def _time_zone_settled(self) -> bool: @@ -507,6 +656,8 @@ async def stop(self) -> None: self._flush_task, self._maintenance_task, self._history_task, + self._pv_task, + self._accuracy_task, ) for task in (*control_tasks, *storage_tasks): if task is not None: @@ -517,6 +668,7 @@ async def stop(self) -> None: await task self._connect_task = self._prune_task = self._tz_task = None self._flush_task = self._maintenance_task = self._history_task = None + self._pv_task = None # Hardware release comes first: a running downsample or retention statement keeps the # (shielded) rollback of its cancelled task waiting, and a Pi can need seconds for it. # The storage tasks are awaited only after the devices are released. @@ -527,6 +679,11 @@ async def stop(self) -> None: if task is not None: with contextlib.suppress(asyncio.CancelledError, Exception): await task + late, self._accuracy_task = self._accuracy_task, None # maybe scheduled while stopping + if late is not None: + late.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await late await self.catalogue.stop() await self.connection.close() await self.recorder.flush() @@ -556,6 +713,8 @@ def _notify_control(self) -> None: def _on_entities(self, update: EntityUpdate) -> None: self.cache.ingest(update) # the cache first: the stack reads it self._last_update = self.real_clock() + if self.pv_service is not None: + self.pv_service.on_entities(update.changed) if self.stack is not None: self.stack.on_update(update) self._notify() @@ -570,6 +729,8 @@ def _on_state(self, state: ConnectionState) -> None: and self._tz_task is None ): self._tz_task = asyncio.ensure_future(self._prepare_control()) + if state is ConnectionState.CONNECTED and self.pv_service is not None: + self.pv_service.wake() # (re)connected: refresh the forecasts now, not in 30 min self._notify() for listener in list(self._health_listeners): try: @@ -581,6 +742,8 @@ async def _resubscribe(self) -> None: refs = list(self.site.entity_refs()) if self.site else [] if self.devices is not None: refs.extend(self.devices.entity_refs()) + if self.forecast is not None: + refs.extend(self.forecast.entity_refs()) ids = list(dict.fromkeys(r.entity_id for r in refs)) try: await self.connection.subscribe_entities(ids) @@ -702,10 +865,15 @@ async def _apply_stored( self.goals = stored if self.stack is not None: self.stack.set_goals(stored) - if name in ("site", "devices"): + elif name == "forecast": + assert isinstance(stored, ForecastConfig) + self.forecast = stored + if name in ("site", "devices", "forecast"): await self._resubscribe() if (self.site, self.devices, self.region) != before: await self._rebuild_stack() + if name == "forecast": + await self._rebuild_pv_service() if name == "devices": await self._sync_modes() @@ -899,8 +1067,8 @@ async def section_history(self, name: str) -> list[HistoryEntry]: async def restore_section(self, name: str, entry_id: int) -> BaseModel: """Restore a history entry; live sections are reloaded like after a normal save.""" restored = await self.config.restore(name, entry_id) - if name in ("site", "devices", "region", "goals", "operation"): - await self._reload_live_sections(resubscribe=name in ("site", "devices")) + if name in ("site", "devices", "region", "goals", "operation", "forecast"): + await self._reload_live_sections(resubscribe=name in ("site", "devices", "forecast")) self._notify() return restored @@ -970,6 +1138,7 @@ async def import_config(self, data: object) -> list[str]: async def _reload_live_sections(self, *, resubscribe: bool) -> None: """After a restore or import: re-read what the control layer runs on and apply it.""" before = (self.site, self.devices, self.region) + forecast_before = self.forecast await self._load_live_sections() if resubscribe: await self._resubscribe() @@ -977,6 +1146,8 @@ async def _reload_live_sections(self, *, resubscribe: bool) -> None: await self._rebuild_stack() elif self.stack is not None and self.goals is not None: self.stack.set_goals(self.goals) + if self.forecast != forecast_before: + await self._rebuild_pv_service() await self._sync_modes() def add_change_listener(self, listener: Callable[[], None]) -> Callable[[], None]: diff --git a/suncore/core/config.py b/suncore/core/config.py index e9d7cfb..18a5109 100644 --- a/suncore/core/config.py +++ b/suncore/core/config.py @@ -212,6 +212,12 @@ class ForecastSourceConfig(_Frozen): """Role of the template (e.g. ``forecast_today``) to entity.""" weight_mode: Literal["auto", "manual"] = "auto" weight: Annotated[float, Field(gt=0, le=1)] | None = None + config_entry_id: ( + Annotated[str, StringConstraints(pattern=r"^[A-Za-z0-9_.:-]{1,64}$")] | None + ) = None + """Template ``series.source: energy_solar_forecast`` (Forecast.Solar): the config entry id + under which ``energy/solar_forecast`` lists the integration (the wizard offers the ids of + ``energy/get_prefs`` ``config_entry_solar_forecast``). ``None`` = the only entry there is.""" @model_validator(mode="after") def _weight(self) -> Self: diff --git a/suncore/forecast/accuracy.py b/suncore/forecast/accuracy.py new file mode 100644 index 0000000..13c5db3 --- /dev/null +++ b/suncore/forecast/accuracy.py @@ -0,0 +1,372 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Accuracy bookkeeping of the PV forecast sources (Phase 4 WP5). + +For every source the forecast that was *issued* ahead of time is compared with the measured PV +power (``site.pv_w``, 15-minute means in ``timeseries.db``), per horizon bucket: + +======== ============== ================================================================== +bucket lead time forecast used for a slot +======== ============== ================================================================== +0-6 h 0 to < 6 h the newest issue at or before the slot start +6-24 h 6 to < 24 h the newest issue at or before ``slot start - 6 h`` +24-48 h 24 to < 48 h the newest issue at or before ``slot start - 24 h`` +======== ============== ================================================================== + +An issue only counts for a bucket if its lead time lies inside it (an issue that is older than +the bucket's upper edge is skipped, not blamed). The forecast is brought to the 15-minute grid +by a time-weighted mean of the source's power (energy preserving); intervals the forecast does +not cover completely are skipped. + +**Sign convention (one for all of SunCore):** ``error = actual - forecast``; ``bias_w`` is the +mean error, so a positive bias means the source under-forecasts. ``mae_w`` is the mean absolute +error, ``n`` the number of 15-minute intervals. + +Only **daylight** intervals count: the forecast or the measurement is above 0 W. A night of +exact zeros would otherwise dilute the MAE and the bias by about half and make every source +look better than it is during the day. Also excluded: intervals without a measurement, and +intervals flagged by the ``is_clipped`` hook (an +inverter or export limit makes the measured PV smaller than what the sun gave, which would +read as a forecast error; :mod:`suncore.forecast.pv.clipping` detects them and the nightly job +:func:`suncore.forecast.pv.learning.run_nightly` passes the hook - the default flags nothing). +""" + +from __future__ import annotations + +import json +import time +from collections.abc import Callable, Iterable, Mapping, Sequence +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import Any, Final, Literal, Protocol + +import structlog + +from suncore.core.forecast import ACCURACY_MIN_SAMPLES, AccuracyRecord, HorizonBucket +from suncore.core.measurements import SITE_PV_W +from suncore.core.series import ( + PLANNING_STEP, + Grid, + QuantileSeries, + SeriesUnit, + TimeRange, + TimeSeries, + floor_to_step, + resample_power, +) +from suncore.forecast.views import AccuracySourceView, ForecastAccuracyView + +log = structlog.get_logger(__name__) + +WINDOW_DAYS: Final = 14 +"""Look-back of the nightly statistics (matches the 14-day inverse-MAE blending weights).""" + +BUCKETS: Final[tuple[tuple[HorizonBucket, timedelta, timedelta], ...]] = ( + (HorizonBucket.H0_6, timedelta(0), timedelta(hours=6)), + (HorizonBucket.H6_24, timedelta(hours=6), timedelta(hours=24)), + (HorizonBucket.H24_48, timedelta(hours=24), timedelta(hours=48)), +) +"""``(bucket, lower edge, upper edge)`` of the lead time.""" + +ClipFilter = Callable[[datetime], bool] +"""``is_clipped(slot_start)``: whether the measured PV of that quarter hour was limited.""" + +SCHEMA_VERSION: Final = 1 +NS_FORECAST_ACCURACY: Final = "forecast_accuracy" +"""``learned_param`` namespace (equal to ``suncore.storage.learned.NS_FORECAST_ACCURACY``; the +forecast package does not import the storage package).""" + + +def no_clipping(_slot_start: datetime) -> bool: + """The default hook: nothing is flagged.""" + return False + + +class AccuracyData(Protocol): + """The part of ``TimeseriesStore`` the statistics read.""" + + async def read_series( + self, + key: str, + window: TimeRange, + resolution: Literal["1m", "15m"] = "15m", + unit: SeriesUnit | None = None, + *, + include_recent: bool = True, + ) -> TimeSeries: ... + + async def forecast_issues( + self, source_id: str, kind: Literal["pv", "load", "price"], since: datetime | None = None + ) -> list[datetime]: ... + + async def read_forecast( + self, + source_id: str, + kind: Literal["pv", "load", "price"], + issued_at: datetime, + window: TimeRange | None = None, + ) -> QuantileSeries | None: ... + + +class LearnedEntry(Protocol): + """A row of ``learned_param`` as ``LearnedParamStore`` returns it.""" + + @property + def key(self) -> str: ... + + @property + def payload(self) -> dict[str, Any]: ... + + @property + def schema_version(self) -> int: ... + + +class LearnedParams(Protocol): + """The part of ``LearnedParamStore`` the bookkeeping uses.""" + + async def put( + self, namespace: str, key: str, payload: Mapping[str, Any], *, schema_version: int = 1 + ) -> None: ... + + async def in_namespace(self, namespace: str) -> Sequence[LearnedEntry]: ... + + +@dataclass(slots=True) +class _Sums: + abs_error: float = 0.0 + error: float = 0.0 + n: int = 0 + + def add(self, error: float) -> None: + self.abs_error += abs(error) + self.error += error + self.n += 1 + + +def quarter_errors( + forecast_p50: TimeSeries, + actual: TimeSeries, + *, + is_clipped: ClipFilter = no_clipping, +) -> list[float]: + """``actual - forecast`` per 15-minute daylight interval (forecast or measurement above 0 W) + both series know and that is not clipped. + + ``forecast_p50`` is resampled onto the grid of ``actual`` (must be the 15-minute grid). + """ + if not len(actual) or not len(forecast_p50): + return [] + on_grid = resample_power( + forecast_p50, Grid(start=actual.start, step=actual.step, count=len(actual)) + ) + errors: list[float] = [] + for (begin, _, measured), (_, _, predicted) in zip( + actual.slots(), on_grid.slots(), strict=True + ): + if measured is None or predicted is None or is_clipped(begin): + continue + if measured <= 0 and predicted <= 0: + continue # night: both zero says nothing about the forecast + errors.append(measured - predicted) + return errors + + +async def compute_accuracy( + data: AccuracyData, + source_id: str, + now: datetime, + *, + window_days: int = WINDOW_DAYS, + is_clipped: ClipFilter = no_clipping, +) -> list[AccuracyRecord]: + """The records of one source for the last ``window_days`` complete days (buckets with data).""" + end = floor_to_step(now, PLANNING_STEP) + start = end - timedelta(days=window_days) + actual = await data.read_series(SITE_PV_W, TimeRange(start=start, end=end), "15m", SeriesUnit.W) + issues = await data.forecast_issues(source_id, "pv", start - BUCKETS[-1][2]) + sums = {bucket: _Sums() for bucket, _, _ in BUCKETS} + for index, issued in enumerate(issues): + following = issues[index + 1] if index + 1 < len(issues) else None + ranges: list[tuple[HorizonBucket, datetime, datetime]] = [] + for bucket, lower, upper in BUCKETS: + lo = _ceil(max(start, issued + lower)) + upper_edge = min(end, issued + upper) + if following is not None: + upper_edge = min(upper_edge, following + lower) # then a newer issue is judged + hi = _ceil(upper_edge) + if hi > lo: + ranges.append((bucket, lo, hi)) + if not ranges: + continue + # One read per issue over the union of its buckets; a forecast slot may start before + # the first quarter hour and still cover it. + first = min(lo for _, lo, _ in ranges) + last = max(hi for _, _, hi in ranges) + forecast = await data.read_forecast( + source_id, "pv", issued, TimeRange(start=first - timedelta(hours=1), end=last) + ) + if forecast is None: + continue + for bucket, lo, hi in ranges: + measured = _slice(actual, lo, hi) + for error in quarter_errors(forecast.p50, measured, is_clipped=is_clipped): + sums[bucket].add(error) + return [ + AccuracyRecord( + source_id=source_id, + horizon=bucket, + mae_w=total.abs_error / total.n, + bias_w=total.error / total.n, + n=total.n, + ) + for bucket, _, _ in BUCKETS + if (total := sums[bucket]).n > 0 + ] + + +def _ceil(moment: datetime) -> datetime: + """``moment`` rounded up to the 15-minute grid.""" + return floor_to_step(moment + PLANNING_STEP - timedelta(microseconds=1), PLANNING_STEP) + + +def _slice(series: TimeSeries, begin: datetime, end: datetime) -> TimeSeries: + """The part of a series between two grid-aligned instants (clamped to its extent).""" + step = series.step + first = max(0, (begin - series.start) // step) + last = min(len(series), -(-(end - series.start) // step)) + last = max(first, last) + return TimeSeries( + start=series.start + step * first, + step=step, + values=series.values[first:last], + unit=series.unit, + ) + + +# --- persistence ------------------------------------------------------------------------- + + +@dataclass(frozen=True, slots=True) +class StoredAccuracy: + """The nightly result of one source as kept in ``learned_param``.""" + + source_id: str + computed_at: datetime + window_days: int + records: tuple[AccuracyRecord, ...] + + +async def save_accuracy( + learned: LearnedParams, + source_id: str, + records: Sequence[AccuracyRecord], + *, + now: datetime, + window_days: int = WINDOW_DAYS, +) -> None: + """Store the records of one source (replacing the previous night's).""" + payload = { + "computed_at": now.isoformat(), + "window_days": window_days, + "records": [json.loads(r.model_dump_json()) for r in records], + } + await learned.put(NS_FORECAST_ACCURACY, source_id, payload, schema_version=SCHEMA_VERSION) + + +async def load_accuracy( + learned: LearnedParams, source_ids: Iterable[str] | None = None +) -> dict[str, StoredAccuracy]: + """The stored results (all, or only ``source_ids``); an unreadable entry is skipped.""" + wanted = None if source_ids is None else set(source_ids) + out: dict[str, StoredAccuracy] = {} + for param in await learned.in_namespace(NS_FORECAST_ACCURACY): + if wanted is not None and param.key not in wanted: + continue + stored = _decode(param.key, param.payload, param.schema_version) + if stored is not None: + out[param.key] = stored + return out + + +def _decode(source_id: str, payload: Mapping[str, object], version: int) -> StoredAccuracy | None: + if version != SCHEMA_VERSION: + log.warning("forecast.accuracy.unknown_schema", source=source_id, version=version) + return None + try: + raw_records = payload["records"] + assert isinstance(raw_records, list) + records = tuple(AccuracyRecord.model_validate(r) for r in raw_records) + computed = datetime.fromisoformat(str(payload["computed_at"])) + window = int(str(payload["window_days"])) + except (KeyError, ValueError, TypeError, AssertionError): + log.warning("forecast.accuracy.unreadable", source=source_id) + return None + return StoredAccuracy(source_id, computed, window, records) + + +def accuracy_view( + source_ids: Iterable[str], stored: Mapping[str, StoredAccuracy] +) -> ForecastAccuracyView: + """The API read model: every configured source, with its stored records if there are any.""" + sources = [] + for source_id in source_ids: + item = stored.get(source_id) + sources.append( + AccuracySourceView( + source_id=source_id, + computed_at=None if item is None else item.computed_at, + window_days=None if item is None else item.window_days, + records=[] if item is None else list(item.records), + ) + ) + return ForecastAccuracyView(min_samples=ACCURACY_MIN_SAMPLES, sources=sources) + + +async def update_accuracy( + data: AccuracyData, + learned: LearnedParams, + source_ids: Iterable[str], + now: datetime, + *, + window_days: int = WINDOW_DAYS, + is_clipped: ClipFilter = no_clipping, +) -> dict[str, list[AccuracyRecord]]: + """The nightly job: recompute and store every source; one failing source does not stop + the others (it is logged and keeps last night's result).""" + results: dict[str, list[AccuracyRecord]] = {} + for source_id in source_ids: + began = time.perf_counter() # runtime for the log only, never for the statistics + try: + records = await compute_accuracy( + data, source_id, now, window_days=window_days, is_clipped=is_clipped + ) + await save_accuracy(learned, source_id, records, now=now, window_days=window_days) + except Exception: + log.exception("forecast.accuracy.failed", source=source_id) + continue + log.info( + "forecast.accuracy.source", + source=source_id, + records=len(records), + runtime_ms=round((time.perf_counter() - began) * 1000, 1), + ) + results[source_id] = records + return results + + +__all__ = [ + "ACCURACY_MIN_SAMPLES", + "BUCKETS", + "NS_FORECAST_ACCURACY", + "WINDOW_DAYS", + "AccuracyData", + "ClipFilter", + "LearnedParams", + "StoredAccuracy", + "accuracy_view", + "compute_accuracy", + "load_accuracy", + "no_clipping", + "quarter_errors", + "save_accuracy", + "update_accuracy", +] diff --git a/suncore/forecast/pv/__init__.py b/suncore/forecast/pv/__init__.py new file mode 100644 index 0000000..1d80381 --- /dev/null +++ b/suncore/forecast/pv/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +"""PV forecast sources, the refresh service and the accuracy bookkeeping (Phase 4 WP5).""" diff --git a/suncore/forecast/pv/adapter.py b/suncore/forecast/pv/adapter.py new file mode 100644 index 0000000..2f827d5 --- /dev/null +++ b/suncore/forecast/pv/adapter.py @@ -0,0 +1,346 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Template-driven PV forecast source (Phase 4 WP5). + +One :class:`TemplateForecastSource` per configured forecast template. It reads the points the +template names - an attribute of the day sensors, the response of a read-only action, or the +Energy dashboard's ``energy/solar_forecast`` - through an injected :class:`PvDataReader` and +returns a :class:`~suncore.core.series.QuantileSeries` in W on the source's native grid +(Solcast 30 min, Open-Meteo Solar Forecast 15 min, Forecast.Solar as spaced by its keys). The +source does no I/O of its own, never writes, and never raises for missing data: it reports what +it found in :attr:`TemplateForecastSource.last_report`. + +Joining the day sensors: the points of ``today``, ``tomorrow``, ``d3`` ... (in the order of the +template's ``series.roles``) are one series; a day whose sensor is unavailable or empty leaves +a gap of ``None`` slots, never a copy of another day. A sensor whose timestamps are irregular +is dropped as a whole (and reported); the others are kept. + +Terms of use: the sources are Home Assistant integrations the user runs; SunCore calls neither +the Solcast API nor Open-Meteo (the built-in fallback is not shipped in Phase 4). +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from datetime import timedelta +from enum import StrEnum, unique +from typing import Any, Final, Protocol, runtime_checkable + +import structlog + +from suncore.core.series import QuantileSeries, TimeRange +from suncore.core.units import Unit, parse_unit +from suncore.forecast.pv.parsing import ( + ParsedPoints, + RawPoint, + build_quantiles, + crop, + dig, + dig_response, + empty_series, + has_known, + list_points, + mapping_points, + regularize_energy, +) +from suncore.templates.models import ( + ActionSeries, + AttributeSeries, + EnergyDashboardSeries, + ForecastTemplate, +) + +log = structlog.get_logger(__name__) + +ACTION_MIN_INTERVAL_S: Final = 600.0 +"""Rate limit of the fallback action (it uses no Solcast API calls, but stay polite).""" + + +@unique +class FetchStatus(StrEnum): + """What a fetch found (rendered as ``forecast.status.`` in the UI).""" + + OK = "ok" + PARTIAL = "partial" + """Data, but a configured sensor was missing, unavailable or irregular.""" + NO_DATA = "no_data" + NOT_CONFIGURED = "not_configured" + """No entity of the template's roles is configured.""" + NOT_LINKED = "not_linked" + """``energy/solar_forecast`` lists no forecast (not linked in the Energy dashboard).""" + AMBIGUOUS_ENTRY = "ambiguous_entry" + """Several entries and none chosen (``config_entry_id``): SunCore does not guess.""" + ENTRY_NOT_FOUND = "entry_not_found" + """The chosen ``config_entry_id`` is not in the command's result.""" + ERROR = "error" + """Set by the service when a fetch raised.""" + + +@dataclass(frozen=True, slots=True) +class FetchReport: + """Diagnostics of the latest fetch of one source.""" + + status: FetchStatus + route: str = "attribute" + """``attribute``, ``action`` (fallback) or ``energy``.""" + entities_used: int = 0 + entities_missing: tuple[str, ...] = () + skipped_points: int = 0 + dropped_entities: tuple[str, ...] = () + """Sensors whose timestamps were irregular.""" + merged_points: int = 0 + """Off-grid energy points added to their enclosing period (``energy/solar_forecast``).""" + + +@runtime_checkable +class PvDataReader(Protocol): + """What a forecast source may read (implemented in ``suncore.app.adapters``).""" + + def attributes(self, entity_id: str) -> Mapping[str, Any] | None: + """Attributes of an entity's current state; ``None`` if unknown or unavailable.""" + ... + + async def solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + """``energy/solar_forecast``: ``{config_entry_id: {iso timestamp: Wh}}``.""" + ... + + async def action_response( + self, domain: str, service: str, data: Mapping[str, Any], *, min_interval_s: float + ) -> Mapping[str, Any]: + """A read-only action with ``return_response`` (rate limited by the connection).""" + ... + + +@dataclass(slots=True) +class _Collected: + points: list[RawPoint] = field(default_factory=list) + skipped: int = 0 + merged: int = 0 + used: int = 0 + missing: list[str] = field(default_factory=list) + dropped: list[str] = field(default_factory=list) + + +def _try_build( + points: tuple[RawPoint, ...], unit: Unit, step: timedelta, *, missing_is_zero: bool = False +) -> QuantileSeries | None: + try: + return build_quantiles(points, unit, step, missing_is_zero=missing_is_zero) + except ValueError: + return None + + +class TemplateForecastSource: + """A :class:`~suncore.core.forecast.ForecastSource` built from a forecast template.""" + + def __init__( + self, + template: ForecastTemplate, + entities: Mapping[str, str], + reader: PvDataReader, + *, + config_entry_id: str | None = None, + ) -> None: + """``entities`` maps the template's roles to entity ids; ``config_entry_id`` binds a + ``energy_solar_forecast`` template to its entry in ``energy/solar_forecast``.""" + self._template = template + self._entities = dict(entities) + self._reader = reader + self._config_entry_id = config_entry_id + self._step = timedelta(minutes=template.resolution_min) + self.last_report = FetchReport(FetchStatus.NO_DATA) + + @property + def source_id(self) -> str: + return self._template.id + + @property + def template(self) -> ForecastTemplate: + return self._template + + @property + def watched_entities(self) -> frozenset[str]: + """Entities whose change should trigger a refresh.""" + return frozenset(self._entities.values()) + + async def fetch(self, window: TimeRange) -> QuantileSeries: + """The forecast for ``window``: what the source has, ``None`` slots where it has not. + + Shorter than the window if the source ends earlier; an empty series (and a report with + the reason) if it has nothing for the window. + """ + series = self._template.series + if isinstance(series, EnergyDashboardSeries): + result, report = await self._from_energy(series) + else: + result, report = await self._from_series(series, window) + fallback = self._template.fallback_series + if ( + fallback is not None + and report.status is not FetchStatus.NOT_CONFIGURED + and (result is None or not has_known(result)) + ): + action, action_report = await self._from_series(fallback, window) + if action is not None and has_known(action): + result, report = action, action_report + cropped = None if result is None else crop(result, window.start, window.end) + if cropped is None or not has_known(cropped): + keep = ( + FetchStatus.NOT_CONFIGURED, + FetchStatus.NOT_LINKED, + FetchStatus.AMBIGUOUS_ENTRY, + FetchStatus.ENTRY_NOT_FOUND, + ) + status = report.status if report.status in keep else FetchStatus.NO_DATA + self.last_report = FetchReport( + status, + report.route, + report.entities_used, + report.entities_missing, + report.skipped_points, + report.dropped_entities, + report.merged_points, + ) + return empty_series(window.start, self._step if result is None else result.step) + self.last_report = report + return cropped + + # --- routes --------------------------------------------------------------------- + + async def _from_series( + self, series: AttributeSeries | ActionSeries, window: TimeRange + ) -> tuple[QuantileSeries | None, FetchReport]: + unit = parse_unit(series.fields.value_unit) + assert unit is not None # validated by the template + if isinstance(series, AttributeSeries): + return self._from_attributes(series, unit) + return await self._from_action(series, unit, window) + + def _from_attributes( + self, series: AttributeSeries, unit: Unit + ) -> tuple[QuantileSeries | None, FetchReport]: + collected = _Collected() + configured = [role for role in series.roles if role in self._entities] + if not configured: + return None, FetchReport(FetchStatus.NOT_CONFIGURED) + for role in configured: + entity_id = self._entities[role] + attrs = self._reader.attributes(entity_id) + raw = None if attrs is None else attrs.get(series.attribute) + parsed = self._parse(series, dig(raw, series.list_path)) + if not parsed.points: + collected.missing.append(entity_id) + collected.skipped += parsed.skipped + continue + if _try_build(parsed.points, unit, self._step) is None: + collected.dropped.append(entity_id) # irregular timestamps: refuse this sensor + continue + collected.points.extend(parsed.points) + collected.skipped += parsed.skipped + collected.used += 1 + return self._join(collected, unit, "attribute", self._step) + + async def _from_action( + self, series: ActionSeries, unit: Unit, window: TimeRange + ) -> tuple[QuantileSeries | None, FetchReport]: + end = min(window.end, window.start + timedelta(hours=series.max_horizon_h)) + data: dict[str, Any] = {} + # UNVERIFIED: the date-time format Solcast's `query_forecast_data` accepts for + # `start_date_time`/`end_date_time` (read from source as a datetime selector; SunCore sends + # ISO 8601 with the UTC offset, not tested against a running integration). + if series.start_field is not None: + data[series.start_field] = window.start.isoformat() + if series.end_field is not None: + data[series.end_field] = end.isoformat() + response = await self._reader.action_response( + series.domain, series.service, data, min_interval_s=ACTION_MIN_INTERVAL_S + ) + parsed = list_points(dig_response(response, series.response_path), series.fields) + collected = _Collected(points=list(parsed.points), skipped=parsed.skipped, used=1) + if not parsed.points: + collected.used = 0 + return self._join(collected, unit, "action", self._step) + + async def _from_energy( + self, series: EnergyDashboardSeries + ) -> tuple[QuantileSeries | None, FetchReport]: + unit = parse_unit(series.fields.value_unit) + assert unit is not None + entries = await self._reader.solar_forecast() + if not entries: + return None, FetchReport(FetchStatus.NOT_LINKED, "energy") + if self._config_entry_id is not None: + chosen = entries.get(self._config_entry_id) + if chosen is None: + return None, FetchReport(FetchStatus.ENTRY_NOT_FOUND, "energy") + elif len(entries) == 1: + chosen = next(iter(entries.values())) + else: + return None, FetchReport(FetchStatus.AMBIGUOUS_ENTRY, "energy") + parsed = mapping_points(chosen) + regular = regularize_energy(parsed.points, self._step) + if regular.dropped or regular.merged: + log.info( + "forecast.pv.off_grid_points", + source=self.source_id, + dropped=regular.dropped, + merged=regular.merged, + ) + collected = _Collected( + points=list(regular.points), + skipped=parsed.skipped + regular.dropped, + merged=regular.merged, + used=int(bool(regular.points)), + ) + return self._join( + collected, unit, "energy", regular.step, missing_is_zero=series.missing_is_zero + ) + + # --- helpers -------------------------------------------------------------------- + + @staticmethod + def _parse(series: AttributeSeries, raw: object) -> ParsedPoints: + if series.shape == "mapping": + return mapping_points(raw) + return list_points(raw, series.fields) + + @staticmethod + def _join( + collected: _Collected, + unit: Unit, + route: str, + step: timedelta, + *, + missing_is_zero: bool = False, + ) -> tuple[QuantileSeries | None, FetchReport]: + built = ( + _try_build(tuple(collected.points), unit, step, missing_is_zero=missing_is_zero) + if collected.points + else None + ) + if built is None and collected.points: + # Sensors that are regular alone but not together (offset grids): refuse the result. + log.warning("forecast.pv.irregular_join", route=route) + incomplete = bool(collected.missing or collected.dropped) + status = FetchStatus.NO_DATA + if built is not None and has_known(built): + status = FetchStatus.PARTIAL if incomplete else FetchStatus.OK + report = FetchReport( + status, + route, + collected.used, + tuple(collected.missing), + collected.skipped, + tuple(collected.dropped), + collected.merged, + ) + return built, report + + +__all__ = [ + "ACTION_MIN_INTERVAL_S", + "FetchReport", + "FetchStatus", + "PvDataReader", + "TemplateForecastSource", +] diff --git a/suncore/forecast/pv/blend.py b/suncore/forecast/pv/blend.py new file mode 100644 index 0000000..c9a5ee9 --- /dev/null +++ b/suncore/forecast/pv/blend.py @@ -0,0 +1,246 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Blending of several PV forecast sources into one quantile forecast (Phase 4 WP5). + +**Weights** are set per horizon bucket (the lead time of a slot: ``0_6h``, ``6_24h``, +``24_48h``; a slot beyond 48 h uses ``24_48h``): + +* ``weight_mode = auto``: the inverse of the source's MAE over the last 14 days + (:mod:`suncore.forecast.accuracy`, records with at least ``ACCURACY_MIN_SAMPLES`` + quarter hours; the MAE is floored at ``MAE_FLOOR_W``). While **any** auto source lacks such a + record for the bucket, the auto sources share equally (``basis = equal_no_history``) - skill + cannot be compared with one side unknown. +* every auto source keeps at least ``min_weight`` (0.1, or ``0.5 / sources`` when there are more + than five), so a source that has data is never dropped entirely: ``w = f + (share - n f) s`` + where ``s`` is the normalised skill, ``f`` the floor and ``share`` what the auto sources get. +* ``weight_mode = manual``: the configured ``weight`` is the source's share of the total. The + auto sources split the rest (``1 - sum(manual)``, but at least their floors; the manual shares + are scaled down to make room). Only manual sources: their weights are normalised. + +In every slot the weights are **renormalised over the sources that have a value there**. + +**Quantiles.** ``P50 = sum(w_i P50_i)``. The spread combines two parts like the law of total +variance (within-source uncertainty + disagreement between sources): + +* ``d_lo = sqrt(sum_Q w_i (P50_i - P10_i)^2 / sum_Q w_i)`` over the sources ``Q`` that provide + real quantiles in the slot (``P10 < P50`` or ``P90 > P50``), ``d_hi`` alike with ``P90``; 0 + when no source provides quantiles; +* ``s = sqrt(sum w_i (P50_i - P50)^2)``, the weighted spread of the sources' medians; +* ``P10 = max(0, P50 - sqrt(d_lo^2 + (z s)^2))``, ``P90 = P50 + sqrt(d_hi^2 + (z s)^2)`` with + ``z = 1.2816`` (the 90 % quantile of a normal distribution). + +So ``P10 <= P50 <= P90`` always holds, agreeing sources keep their own band, and disagreeing +ones widen it. A slot with a single source passes its values through unchanged. +""" + +from __future__ import annotations + +import math +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from typing import Final, Literal + +from suncore.core.forecast import ACCURACY_MIN_SAMPLES, AccuracyRecord, HorizonBucket +from suncore.core.series import ( + PLANNING_STEP, + Grid, + QuantileSeries, + SeriesUnit, + TimeSeries, + planning_grid, + resample_quantiles, +) + +MIN_WEIGHT: Final = 0.1 +MAE_FLOOR_W: Final = 10.0 +Z90: Final = 1.2815515655446004 +BLEND_HORIZON: Final = timedelta(hours=48) + +WEIGHT_BUCKETS: Final = (HorizonBucket.H0_6, HorizonBucket.H6_24, HorizonBucket.H24_48) +AutoBasis = Literal["inverse_mae", "equal_no_history"] +BlendMethod = Literal["weighted", "single_source"] + + +@dataclass(frozen=True, slots=True) +class BlendInput: + """One source as the blender sees it.""" + + source_id: str + series: QuantileSeries + weight_mode: Literal["auto", "manual"] = "auto" + manual_weight: float | None = None + records: Sequence[AccuracyRecord] = () + + @property + def is_manual(self) -> bool: + return self.weight_mode == "manual" and self.manual_weight is not None + + +@dataclass(frozen=True, slots=True) +class BucketWeights: + """The weights of one horizon bucket before the per-slot renormalisation.""" + + horizon: HorizonBucket + weights: Mapping[str, float] + auto_basis: AutoBasis | None + """How the auto sources were weighted; ``None`` = there is no auto source.""" + manual: bool + """Whether a manual weight took part.""" + + +@dataclass(frozen=True, slots=True) +class BlendResult: + series: QuantileSeries + method: BlendMethod + weights: tuple[BucketWeights, ...] + sources: tuple[str, ...] = field(default=()) + + +def weight_bucket(lead: timedelta) -> HorizonBucket: + """The bucket whose weights apply to a slot ``lead`` after the blend time.""" + bucket = HorizonBucket.of(lead) + return HorizonBucket.H24_48 if bucket is HorizonBucket.H48_PLUS else bucket + + +def _usable_mae(records: Sequence[AccuracyRecord], bucket: HorizonBucket) -> float | None: + for record in records: + if record.horizon is bucket and record.n >= ACCURACY_MIN_SAMPLES: + return record.mae_w + return None + + +def source_weights( + inputs: Sequence[BlendInput], + bucket: HorizonBucket, + *, + min_weight: float = MIN_WEIGHT, + mae_floor_w: float = MAE_FLOOR_W, +) -> BucketWeights: + """The weights of ``inputs`` for one bucket (sum 1; module doc).""" + if not inputs: + return BucketWeights(bucket, {}, None, manual=False) + manual = [i for i in inputs if i.is_manual] + auto = [i for i in inputs if not i.is_manual] + floor = min(min_weight, 0.5 / len(inputs)) + weights: dict[str, float] = {} + manual_sum = sum(i.manual_weight or 0.0 for i in manual) + if not auto: + for item in manual: + weights[item.source_id] = (item.manual_weight or 0.0) / manual_sum + return BucketWeights(bucket, weights, None, manual=True) + auto_share = 1.0 if not manual else min(1.0, max(1.0 - manual_sum, floor * len(auto))) + for item in manual: + weights[item.source_id] = (item.manual_weight or 0.0) * (1.0 - auto_share) / manual_sum + maes = [_usable_mae(i.records, bucket) for i in auto] + basis: AutoBasis + if all(m is not None for m in maes): + inverse = [1.0 / max(m or 0.0, mae_floor_w) for m in maes] + total = sum(inverse) + skill = [v / total for v in inverse] + basis = "inverse_mae" + else: + skill = [1.0 / len(auto)] * len(auto) + basis = "equal_no_history" + spare = auto_share - floor * len(auto) + for item, s in zip(auto, skill, strict=True): + weights[item.source_id] = floor + spare * s + return BucketWeights(bucket, weights, basis, manual=bool(manual)) + + +SlotValues = tuple[float, float | None, float, float | None] +"""``(weight, p10, p50, p90)`` of one source in one slot.""" + + +def blend_slot(values: Sequence[SlotValues]) -> tuple[float, float, float] | None: + """``(P10, P50, P90)`` of one slot from the sources that have a P50 there (module doc).""" + if not values: + return None + if len(values) == 1: + _, lo, mid, hi = values[0] + return (mid if lo is None else lo, mid, mid if hi is None else hi) + total = sum(v[0] for v in values) + if total <= 0: + return None + p50 = sum(w * mid for w, _, mid, _ in values) / total + spread2 = sum(w * (mid - p50) ** 2 for w, _, mid, _ in values) / total + q_weight = 0.0 + lo2 = hi2 = 0.0 + for w, lo, mid, hi in values: + if lo is None or hi is None or (lo >= mid and hi <= mid): + continue + q_weight += w + lo2 += w * max(0.0, mid - lo) ** 2 + hi2 += w * max(0.0, hi - mid) ** 2 + if q_weight > 0: + lo2 /= q_weight + hi2 /= q_weight + disagreement2 = (Z90**2) * spread2 + p10 = p50 - math.sqrt(lo2 + disagreement2) + if p50 >= 0: + p10 = max(0.0, p10) + p90 = p50 + math.sqrt(hi2 + disagreement2) + return (min(p10, p50), p50, max(p90, p50)) + + +def blend( + inputs: Sequence[BlendInput], + now: datetime, + *, + horizon: timedelta = BLEND_HORIZON, + grid: Grid | None = None, + min_weight: float = MIN_WEIGHT, +) -> BlendResult | None: + """The blended forecast on the 15-minute planning grid from ``now`` (``None``: no data).""" + target = grid or planning_grid(now, horizon, PLANNING_STEP) + on_grid = [(i, resample_quantiles(i.series, target)) for i in inputs] + present = [(i, q) for i, q in on_grid if any(v is not None for v in q.p50.values)] + if not present: + return None + used = [i for i, _ in present] + per_bucket = {b: source_weights(used, b, min_weight=min_weight) for b in WEIGHT_BUCKETS} + p10: list[float | None] = [] + p50: list[float | None] = [] + p90: list[float | None] = [] + for index, begin in enumerate(target.starts()): + weights = per_bucket[weight_bucket(begin - now)].weights + values: list[SlotValues] = [] + for item, series in present: + mid = series.p50.values[index] + if mid is None: + continue + values.append( + (weights[item.source_id], series.p10.values[index], mid, series.p90.values[index]) + ) + slot = blend_slot(values) + p10.append(None if slot is None else slot[0]) + p50.append(None if slot is None else slot[1]) + p90.append(None if slot is None else slot[2]) + + def part(values: list[float | None]) -> TimeSeries: + return TimeSeries( + start=target.start, step=target.step, values=tuple(values), unit=SeriesUnit.W + ) + + return BlendResult( + series=QuantileSeries(p10=part(p10), p50=part(p50), p90=part(p90)), + method="single_source" if len(present) == 1 else "weighted", + weights=tuple(per_bucket[b] for b in WEIGHT_BUCKETS), + sources=tuple(i.source_id for i in used), + ) + + +__all__ = [ + "BLEND_HORIZON", + "MAE_FLOOR_W", + "MIN_WEIGHT", + "WEIGHT_BUCKETS", + "AutoBasis", + "BlendInput", + "BlendMethod", + "BlendResult", + "BucketWeights", + "blend", + "blend_slot", + "source_weights", + "weight_bucket", +] diff --git a/suncore/forecast/pv/calibration.py b/suncore/forecast/pv/calibration.py new file mode 100644 index 0000000..801d9ad --- /dev/null +++ b/suncore/forecast/pv/calibration.py @@ -0,0 +1,374 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Bias calibration of a PV forecast source (Phase 4 WP5). + +Every source gets a table of **multiplicative factors** ``actual / forecast`` per cell +``hour of day x season x weather class``: + +* *hour* - the local hour (site time zone) of the slot start; +* *season* - meteorological month groups ``djf``, ``mam``, ``jja``, ``son`` (the key is the + month group, so it works on both hemispheres); +* *weather class* - the clear-sky index of the source's own P50 + (:mod:`suncore.forecast.pv.clearsky`). + +**Learning** (once a night, one complete local day at a time): the source's issue that was +current at the start of the day (lead 0-24 h; what the plan for that day relied on) is put on the +15-minute grid and compared with the measured PV. A quarter hour counts when the forecast is at +least ``MIN_SAMPLE_W``, the measurement is known and the slot is **not clipped**. A cell keeps +energy sums (``forecast_wh``, ``actual_wh``) and a sample count ``n``; before a day's samples are +added, the cell's sums are multiplied by ``DECAY`` (an older season fades slowly; a cell that is +not visited keeps its values, so last year's winter is still there next winter). + +**Factor** of a cell: ``raw = actual_wh / forecast_wh`` bounded to ``FACTOR_BOUNDS``, shrunk to 1 +with few samples: ``f = 1 + n / (n + SHRINK_K) * (raw - 1)``. A cell without samples has +``f = 1``. The same factor scales P10, P50 and P90, so their order is kept. + +**Activation**: the table is learnt (and stored) from the first day on, but applied only when +``days_with_data >= min_history_days`` (``forecast.learning.min_history_days``, default 30, the +maintainer's answer). Before that the API shows the calibrated forecast next to the uncalibrated +one for comparison, and the planner gets the uncalibrated one. + +The table is stored in ``learned_param`` namespace ``forecast_bias``, key = source id, through +the :class:`~suncore.forecast.accuracy.LearnedParams` protocol. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field, replace +from datetime import datetime, timedelta, tzinfo +from typing import Any, Final, Literal + +import structlog + +from suncore.core.series import Grid, QuantileSeries, TimeSeries, resample_power +from suncore.forecast.accuracy import LearnedParams +from suncore.forecast.pv.clearsky import ( + ClassifierMode, + PlantGeometry, + WeatherClass, + classifier_mode, + classify_slots, + daily_peaks, + peak_reference, +) + +log = structlog.get_logger(__name__) + +NS_FORECAST_BIAS: Final = "forecast_bias" +"""``learned_param`` namespace (equal to ``suncore.storage.learned.NS_FORECAST_BIAS``).""" +SCHEMA_VERSION: Final = 1 + +FACTOR_BOUNDS: Final = (0.5, 1.5) +SHRINK_K: Final = 16.0 +"""Quarter hours at which a cell's factor goes half-way from 1 to its raw ratio.""" +DECAY: Final = 0.98 +"""Per learned day with samples in the cell (half-life about 34 such days).""" +PEAK_HISTORY_DAYS: Final = 365 +MIN_SAMPLE_W: Final = 50.0 +"""Slots with a smaller forecast (dawn, dusk) carry no usable ratio.""" + +SEASONS: Final[Mapping[int, str]] = { + 12: "djf", + 1: "djf", + 2: "djf", + 3: "mam", + 4: "mam", + 5: "mam", + 6: "jja", + 7: "jja", + 8: "jja", + 9: "son", + 10: "son", + 11: "son", +} + +CalibrationReason = Literal["active", "collecting", "no_history", "classifier_changed"] +"""``active``: applied. ``collecting``: fewer than ``min_history_days`` days of data. +``no_history``: nothing learnt yet. ``classifier_changed``: the table was learnt with the other +weather classifier (the location became unknown); it waits instead of being misapplied.""" + + +def season(local: datetime) -> str: + return SEASONS[local.month] + + +def cell_key(slot_start: datetime, weather: WeatherClass, tz: tzinfo) -> str: + local = slot_start.astimezone(tz) + return f"{local.hour:02d}.{season(local)}.{weather.value}" + + +@dataclass(frozen=True, slots=True) +class CellStats: + n: float = 0.0 + forecast_wh: float = 0.0 + actual_wh: float = 0.0 + + +def _bounded(value: float, bounds: tuple[float, float] = FACTOR_BOUNDS) -> float: + return min(bounds[1], max(bounds[0], value)) + + +def cell_factor( + stats: CellStats | None, + *, + shrink_k: float = SHRINK_K, + bounds: tuple[float, float] = FACTOR_BOUNDS, +) -> float: + """The cell's factor: bounded raw ratio, shrunk to 1 by ``n / (n + k)`` (module doc).""" + if stats is None or stats.n <= 0 or stats.forecast_wh <= 0: + return 1.0 + raw = _bounded(stats.actual_wh / stats.forecast_wh, bounds) + return _bounded(1.0 + stats.n / (stats.n + shrink_k) * (raw - 1.0), bounds) + + +@dataclass(frozen=True, slots=True) +class BiasTable: + """The learnt state of one source.""" + + source_id: str + mode: ClassifierMode + cells: Mapping[str, CellStats] = field(default_factory=dict) + days_with_data: int = 0 + learned_until: datetime | None = None + """End of the last day that was processed (with or without samples).""" + daily_peaks: tuple[float, ...] = () + """Forecast peak of every learnt day, newest last (at most ``PEAK_HISTORY_DAYS``).""" + + @property + def peak_reference_w(self) -> float | None: + """Reference of the ``daily_peak`` classifier: 95th percentile of ``daily_peaks``.""" + return peak_reference(self.daily_peaks) + + @property + def samples(self) -> float: + return sum(c.n for c in self.cells.values()) + + +def new_table(source_id: str, plant: PlantGeometry | None) -> BiasTable: + return BiasTable(source_id=source_id, mode=classifier_mode(plant)) + + +@dataclass(frozen=True, slots=True) +class CalibrationState: + """Whether a source's calibration is applied and why (API read model input).""" + + source_id: str + active: bool + reason: CalibrationReason + days_with_data: int + min_history_days: int + cells: int + factor_min: float | None + factor_max: float | None + + +def calibration_state( + source_id: str, + table: BiasTable | None, + *, + plant: PlantGeometry | None, + min_history_days: int, +) -> CalibrationState: + if table is None or not table.cells: + return CalibrationState(source_id, False, "no_history", 0, min_history_days, 0, None, None) + factors = [cell_factor(c) for c in table.cells.values() if c.n > 0] + reason: CalibrationReason + if table.mode != classifier_mode(plant): + reason = "classifier_changed" + elif table.days_with_data < min_history_days: + reason = "collecting" + else: + reason = "active" + return CalibrationState( + source_id=source_id, + active=reason == "active", + reason=reason, + days_with_data=table.days_with_data, + min_history_days=min_history_days, + cells=len(factors), + factor_min=min(factors) if factors else None, + factor_max=max(factors) if factors else None, + ) + + +def slot_factors( + forecast_p50: TimeSeries, table: BiasTable, *, plant: PlantGeometry | None, tz: tzinfo +) -> list[float]: + """The factor of every slot of ``forecast_p50`` (1 where there is no class or no data).""" + if table.mode != classifier_mode(plant): + return [1.0] * len(forecast_p50) + classes = classify_slots( + forecast_p50, plant=plant, tz=tz, peak_reference_w=table.peak_reference_w + ) + out: list[float] = [] + for (begin, _, _), weather in zip(forecast_p50.slots(), classes, strict=True): + out.append( + 1.0 if weather is None else cell_factor(table.cells.get(cell_key(begin, weather, tz))) + ) + return out + + +def calibrate( + series: QuantileSeries, table: BiasTable, *, plant: PlantGeometry | None, tz: tzinfo +) -> QuantileSeries: + """``series`` with every quantile scaled by its slot's factor (order preserved).""" + factors = slot_factors(series.p50, table, plant=plant, tz=tz) + + def scaled(part: TimeSeries) -> TimeSeries: + values = tuple( + None if v is None else v * f for v, f in zip(part.values, factors, strict=True) + ) + return TimeSeries(start=part.start, step=part.step, values=values, unit=part.unit) + + return QuantileSeries(p10=scaled(series.p10), p50=scaled(series.p50), p90=scaled(series.p90)) + + +def learn_day( + table: BiasTable, + forecast_p50: TimeSeries, + actual: TimeSeries, + *, + plant: PlantGeometry | None, + tz: tzinfo, + day_end: datetime, + is_clipped: Callable[[datetime], bool] = lambda _slot: False, + decay: float = DECAY, + min_sample_w: float = MIN_SAMPLE_W, +) -> BiasTable: + """``table`` with one more day learnt (``actual`` on the 15-minute grid of that day). + + A table learnt with the other classifier is reset first when the current one is + ``clear_sky`` (the location became known: the better classifier wins). The other way round + (the location is unknown tonight) the table is returned unchanged, so a night without Home + Assistant does not throw away what was learnt. + """ + mode = classifier_mode(plant) + if table.mode != mode: + if mode == "daily_peak": + return table + log.info("forecast.bias.classifier_reset", source=table.source_id) + table = BiasTable(source_id=table.source_id, mode=mode) + if not len(actual) or not len(forecast_p50): + return replace(table, learned_until=day_end) + grid = Grid(start=actual.start, step=actual.step, count=len(actual)) + forecast = resample_power(forecast_p50, grid) + classes = classify_slots(forecast, plant=plant, tz=tz, peak_reference_w=table.peak_reference_w) + hours = actual.step / timedelta(hours=1) + day: dict[str, CellStats] = {} + for (begin, _, measured), predicted, weather in zip( + actual.slots(), forecast.values, classes, strict=True + ): + if ( + measured is None + or predicted is None + or weather is None + or predicted < min_sample_w + or is_clipped(begin) + ): + continue + key = cell_key(begin, weather, tz) + old = day.get(key, CellStats()) + day[key] = CellStats( + n=old.n + 1, + forecast_wh=old.forecast_wh + predicted * hours, + actual_wh=old.actual_wh + max(0.0, measured) * hours, + ) + cells = dict(table.cells) + for key, add in day.items(): + old = cells.get(key, CellStats()) + cells[key] = CellStats( + n=old.n * decay + add.n, + forecast_wh=old.forecast_wh * decay + add.forecast_wh, + actual_wh=old.actual_wh * decay + add.actual_wh, + ) + new_peaks = tuple(p for p in daily_peaks(forecast, tz).values() if p > 0) + history = (*table.daily_peaks, *new_peaks)[-PEAK_HISTORY_DAYS:] + return BiasTable( + source_id=table.source_id, + mode=mode, + cells=cells, + days_with_data=table.days_with_data + (1 if day else 0), + learned_until=day_end, + daily_peaks=history, + ) + + +# --- persistence ------------------------------------------------------------------------- + + +def encode(table: BiasTable) -> dict[str, Any]: + return { + "mode": table.mode, + "days_with_data": table.days_with_data, + "learned_until": None if table.learned_until is None else table.learned_until.isoformat(), + "daily_peaks": list(table.daily_peaks), + "cells": {k: [c.n, c.forecast_wh, c.actual_wh] for k, c in sorted(table.cells.items())}, + } + + +def decode(source_id: str, payload: Mapping[str, Any], version: int) -> BiasTable | None: + if version != SCHEMA_VERSION: + log.warning("forecast.bias.unknown_schema", source=source_id, version=version) + return None + try: + mode = payload["mode"] + if mode not in ("clear_sky", "daily_peak"): + raise ValueError(mode) + until_raw = payload.get("learned_until") + until = None if until_raw is None else datetime.fromisoformat(str(until_raw)) + peaks_raw = payload.get("daily_peaks", []) + if not isinstance(peaks_raw, list): + raise TypeError("daily_peaks") + peaks = tuple(float(v) for v in peaks_raw) + raw_cells = payload["cells"] + if not isinstance(raw_cells, Mapping): + raise TypeError("cells") + cells: dict[str, CellStats] = {} + for key, value in raw_cells.items(): + n, f_wh, a_wh = (float(v) for v in value) + cells[str(key)] = CellStats(n=n, forecast_wh=f_wh, actual_wh=a_wh) + days = int(payload["days_with_data"]) + except (KeyError, ValueError, TypeError): + log.warning("forecast.bias.unreadable", source=source_id) + return None + return BiasTable(source_id, mode, cells, days, until, peaks) + + +async def save_bias(learned: LearnedParams, table: BiasTable) -> None: + await learned.put( + NS_FORECAST_BIAS, table.source_id, encode(table), schema_version=SCHEMA_VERSION + ) + + +async def load_bias(learned: LearnedParams) -> dict[str, BiasTable]: + out: dict[str, BiasTable] = {} + for param in await learned.in_namespace(NS_FORECAST_BIAS): + table = decode(param.key, param.payload, param.schema_version) + if table is not None: + out[param.key] = table + return out + + +__all__ = [ + "DECAY", + "FACTOR_BOUNDS", + "MIN_SAMPLE_W", + "NS_FORECAST_BIAS", + "SHRINK_K", + "BiasTable", + "CalibrationReason", + "CalibrationState", + "CellStats", + "calibrate", + "calibration_state", + "cell_factor", + "cell_key", + "decode", + "encode", + "learn_day", + "load_bias", + "new_table", + "save_bias", + "season", + "slot_factors", +] diff --git a/suncore/forecast/pv/clearsky.py b/suncore/forecast/pv/clearsky.py new file mode 100644 index 0000000..ad3c0c5 --- /dev/null +++ b/suncore/forecast/pv/clearsky.py @@ -0,0 +1,267 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Clear-sky reference of the plant and the weather class of a forecast slot (Phase 4 WP5). + +The bias calibration needs a *weather class* per slot - how cloudy the forecast itself thinks the +slot is. It is the **clear-sky index** ``kt = forecast / clear-sky estimate`` of the forecast's +own P50, bucketed into :class:`WeatherClass`. Two ways to get the clear-sky estimate: + +``clear_sky`` (location and every array's ``kwp`` known) + A simple sun-position model (declination, equation of time, hour angle; no refraction) gives + the sun's elevation and azimuth; the Haurwitz clear-sky model gives the global horizontal + irradiance ``GHI = 1098 cos(Z) exp(-0.057 / cos(Z))`` W/m2, split into 10 % diffuse and the + rest direct, transposed onto every array (azimuth/tilt; missing values default to an + equator-facing 30 deg plane) with an isotropic sky and 20 % ground albedo. The plant's + clear-sky power is ``sum(kwp * POA)`` (W, 1 kWp yields 1 kW at 1000 W/m2), without losses: + a clear day therefore reads ``kt`` of about 0.75-0.95. The location comes from Home + Assistant's ``get_config`` (``latitude``/``longitude``), the arrays from ``site.pv``. + +``daily_peak`` (fallback: no location or an array without ``kwp``) + No sun model is possible, so the class is a **day-level** index: the forecast's peak of the + local day divided by a reference peak: the 95th percentile of the daily forecast peaks the + bias table has stored (up to a year; a percentile, not the maximum, so one absurd forecast + does not push every later day into ``overcast``), or - without stored peaks - of the + series' own daily peaks. All daytime slots of a day share the class; a slot is daytime when + the forecast exceeds 2 % of the reference peak. + +The exact numbers matter little: calibration only needs the class to be a *consistent* +function of the forecast, because the same function sorts the learning samples and the slots +the factors are applied to. A systematic offset of the model (an east-west roof, a shaded +plane) shifts slots between classes the same way in both. +""" + +from __future__ import annotations + +import math +from collections.abc import Sequence +from dataclasses import dataclass +from datetime import UTC, date, datetime, timedelta, tzinfo +from enum import StrEnum, unique +from typing import Final, Literal + +from suncore.core.series import TimeSeries +from suncore.core.site import PvArray + +ClassifierMode = Literal["clear_sky", "daily_peak"] + +OVERCAST_BELOW: Final = 0.3 +"""``kt`` below this is ``overcast``.""" +CLEAR_FROM: Final = 0.65 +"""``kt`` from this on is ``clear``; in between ``mixed``.""" +TWILIGHT_SHARE: Final = 0.03 +"""Clear-sky power below this share of the installed kWp is twilight: no class.""" +DAYTIME_SHARE: Final = 0.02 +"""``daily_peak`` mode: a forecast below this share of the reference peak is not daytime.""" +DIFFUSE_SHARE: Final = 0.10 +ALBEDO: Final = 0.20 +DEFAULT_TILT_DEG: Final = 30.0 + + +@unique +class WeatherClass(StrEnum): + """Cloudiness of a slot as the forecast sees it (clear-sky index bucket).""" + + OVERCAST = "overcast" + MIXED = "mixed" + CLEAR = "clear" + + +def classify_kt(kt: float) -> WeatherClass: + if kt < OVERCAST_BELOW: + return WeatherClass.OVERCAST + if kt < CLEAR_FROM: + return WeatherClass.MIXED + return WeatherClass.CLEAR + + +@dataclass(frozen=True, slots=True) +class ArrayGeometry: + kwp: float + azimuth_deg: float + """0 = north, 90 = east, 180 = south.""" + tilt_deg: float + + +@dataclass(frozen=True, slots=True) +class PlantGeometry: + latitude: float + longitude: float + arrays: tuple[ArrayGeometry, ...] + + @property + def total_kwp(self) -> float: + return sum(a.kwp for a in self.arrays) + + +def plant_geometry( + latitude: float | None, longitude: float | None, arrays: Sequence[PvArray] +) -> PlantGeometry | None: + """The geometry for the ``clear_sky`` mode, or ``None`` (fallback ``daily_peak``). + + ``None`` when the location is unknown, there is no array or an array has no ``kwp``. + """ + if latitude is None or longitude is None or not arrays: + return None + if not (-90 <= latitude <= 90 and -180 <= longitude <= 180): + return None + default_azimuth = 180.0 if latitude >= 0 else 0.0 + out: list[ArrayGeometry] = [] + for array in arrays: + if array.kwp is None: + return None + out.append( + ArrayGeometry( + kwp=array.kwp, + azimuth_deg=default_azimuth if array.azimuth_deg is None else array.azimuth_deg, + tilt_deg=DEFAULT_TILT_DEG if array.tilt_deg is None else array.tilt_deg, + ) + ) + return PlantGeometry(latitude=latitude, longitude=longitude, arrays=tuple(out)) + + +def sun_position(when: datetime, latitude: float, longitude: float) -> tuple[float, float]: + """``(elevation, azimuth)`` in radians; azimuth from north, clockwise (90 deg = east).""" + utc = when.astimezone(UTC) + day = utc.timetuple().tm_yday + hours = utc.hour + utc.minute / 60 + utc.second / 3600 + gamma = 2 * math.pi / 365 * (day - 1 + (hours - 12) / 24) + declination = ( + 0.006918 + - 0.399912 * math.cos(gamma) + + 0.070257 * math.sin(gamma) + - 0.006758 * math.cos(2 * gamma) + + 0.000907 * math.sin(2 * gamma) + - 0.002697 * math.cos(3 * gamma) + + 0.00148 * math.sin(3 * gamma) + ) + eq_time_min = 229.18 * ( + 0.000075 + + 0.001868 * math.cos(gamma) + - 0.032077 * math.sin(gamma) + - 0.014615 * math.cos(2 * gamma) + - 0.040849 * math.sin(2 * gamma) + ) + solar_min = hours * 60 + eq_time_min + 4 * longitude + hour_angle = math.radians(solar_min / 4 - 180) + lat = math.radians(latitude) + sin_el = math.sin(lat) * math.sin(declination) + math.cos(lat) * math.cos( + declination + ) * math.cos(hour_angle) + sin_el = min(1.0, max(-1.0, sin_el)) + elevation = math.asin(sin_el) + cos_el = math.cos(elevation) + if cos_el < 1e-9 or abs(math.cos(lat)) < 1e-9: + return elevation, math.pi + cos_az = (math.sin(declination) - sin_el * math.sin(lat)) / (cos_el * math.cos(lat)) + azimuth = math.acos(min(1.0, max(-1.0, cos_az))) + if hour_angle > 0: + azimuth = 2 * math.pi - azimuth + return elevation, azimuth + + +def clear_sky_power_w(plant: PlantGeometry, when: datetime) -> float: + """Clear-sky power of the plant at one instant (W, no losses).""" + elevation, sun_az = sun_position(when, plant.latitude, plant.longitude) + cos_z = math.sin(elevation) + if cos_z <= 0.01: + return 0.0 + ghi = 1098.0 * cos_z * math.exp(-0.057 / cos_z) + dhi = DIFFUSE_SHARE * ghi + dni = (ghi - dhi) / cos_z + sin_z = math.cos(elevation) + total = 0.0 + for array in plant.arrays: + beta = math.radians(array.tilt_deg) + cos_inc = cos_z * math.cos(beta) + sin_z * math.sin(beta) * math.cos( + sun_az - math.radians(array.azimuth_deg) + ) + poa = ( + dni * max(0.0, cos_inc) + + dhi * (1 + math.cos(beta)) / 2 + + ALBEDO * ghi * (1 - math.cos(beta)) / 2 + ) + total += array.kwp * poa + return total + + +def clear_sky_slot_w(plant: PlantGeometry, start: datetime, step: timedelta) -> float: + """Mean clear-sky power over a slot (three sample points).""" + points = (start + step * (k / 6) for k in (1, 3, 5)) + return sum(clear_sky_power_w(plant, p) for p in points) / 3 + + +PEAK_PERCENTILE: Final = 0.95 + + +def peak_reference(peaks: Sequence[float]) -> float | None: + """The ``PEAK_PERCENTILE`` (nearest rank) of positive daily peaks, ``None`` without any.""" + ordered = sorted(p for p in peaks if p > 0) + if not ordered: + return None + rank = max(1, math.ceil(PEAK_PERCENTILE * len(ordered))) + return ordered[rank - 1] + + +def daily_peaks(series: TimeSeries, tz: tzinfo) -> dict[date, float]: + """Largest known value per local day.""" + peaks: dict[date, float] = {} + for begin, _, value in series.slots(): + if value is None: + continue + day = begin.astimezone(tz).date() + peaks[day] = max(peaks.get(day, 0.0), value) + return peaks + + +def classify_slots( + forecast: TimeSeries, + *, + plant: PlantGeometry | None, + tz: tzinfo, + peak_reference_w: float | None = None, +) -> list[WeatherClass | None]: + """The weather class of every slot of ``forecast`` (P50, W); ``None`` = night/unknown. + + ``plant`` selects the mode (module doc); ``peak_reference_w`` is the ``daily_peak`` mode's + reference (``None``: the percentile of the series' own daily peaks). + """ + if plant is not None: + twilight = TWILIGHT_SHARE * plant.total_kwp * 1000 + out: list[WeatherClass | None] = [] + for begin, end, value in forecast.slots(): + clear = clear_sky_slot_w(plant, begin, end - begin) + out.append(None if value is None or clear < twilight else classify_kt(value / clear)) + return out + peaks = daily_peaks(forecast, tz) + reference = peak_reference_w or peak_reference(list(peaks.values())) or 0.0 + if reference <= 0: + return [None] * len(forecast) + result: list[WeatherClass | None] = [] + for begin, _, value in forecast.slots(): + if value is None or value < DAYTIME_SHARE * reference: + result.append(None) + continue + result.append(classify_kt(peaks[begin.astimezone(tz).date()] / reference)) + return result + + +def classifier_mode(plant: PlantGeometry | None) -> ClassifierMode: + return "daily_peak" if plant is None else "clear_sky" + + +__all__ = [ + "CLEAR_FROM", + "OVERCAST_BELOW", + "ArrayGeometry", + "ClassifierMode", + "PlantGeometry", + "WeatherClass", + "classifier_mode", + "classify_kt", + "classify_slots", + "clear_sky_power_w", + "clear_sky_slot_w", + "daily_peaks", + "peak_reference", + "plant_geometry", + "sun_position", +] diff --git a/suncore/forecast/pv/clipping.py b/suncore/forecast/pv/clipping.py new file mode 100644 index 0000000..2e05218 --- /dev/null +++ b/suncore/forecast/pv/clipping.py @@ -0,0 +1,171 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Clipping detection: quarter hours whose measured PV was limited (Phase 4 WP5). + +When the inverter's AC rating or the grid's feed-in limit caps the plant, the measured PV is +smaller than what the sun gave. Such a slot would read as a forecast error (a too-high forecast) +and would teach the blending weights and the bias calibration the wrong lesson, so the accuracy +job and the calibration learner skip it (``is_clipped`` hook). Everything here works on the +15-minute means of ``timeseries.db``; values in W, grid + = import (export is negative). + +Two signals, both conservative (a missed clip costs a little accuracy; a false one only drops a +sample): + +**Export limit** (``site.grid.export_limit_w``, or ``export_limit_pct_kwp`` x the installed kWp) + A slot is flagged when the mean grid export lies within ``max(3 % of the limit, 50 W)`` of + the limit **and** the reference forecast is higher than the measured PV (when one is known). + The plateau of an export limit shows in the *grid export*, not in the PV: a limiting inverter + lets the PV follow the house load (PV = limit + load + battery charge), so the export pinned + at the limit is the flat signal, and "PV flat" is not required here. + + **Zero-export plants** (limit 0) are not detected by this rule, only by the plateau rule + below. Decision: with a limit of 0 the grid sits at about 0 W whether the PV is curtailed or + not - a battery charging from PV, or plain self-consumption, keeps it there too - so "export + at the limit while the forecast is higher" would flag every over-forecast and hide exactly + the errors the statistics should see. + +**Inverter plateau** (derived from the data) + SunCore has no inverter AC rating in its configuration (``PvArray.inverter`` is a name), so + none is invented: the cap is the PV level that the data itself repeats. The window's largest + PV value is the cap when on at least ``MIN_CAP_DAYS`` local days the PV stayed for at least + ``MIN_RUN`` consecutive quarter hours within ``PLATEAU_TOLERANCE`` of it - a natural noon + peak is rounded and noisy, a clipping inverter is flat at the same value every sunny day. A + slot is flagged when its PV is within ``CAP_BAND`` of the cap and the reference forecast is + higher (when known). + +The **reference forecast** is the mean of the sources' P50 as they stood at the start of each +day (see :mod:`suncore.forecast.pv.learning`); "higher" means more than ``FORECAST_MARGIN`` +above the measured PV. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from datetime import date, datetime, tzinfo +from typing import Final + +from suncore.core.series import TimeSeries +from suncore.core.site import SiteConfig + +PLATEAU_TOLERANCE: Final = 0.003 +"""Relative spread of a plateau's quarter hours around the cap.""" +MIN_RUN: Final = 4 +"""Consecutive quarter hours at the cap that make a plateau (1 h; a noise-free clear-sky noon +stays within 0.3 % of its peak for about 45 min around the solstice, a clipping inverter for +hours).""" +MIN_CAP_DAYS: Final = 2 +"""Days with a plateau at the same level before it counts as the inverter's cap.""" +CAP_BAND: Final = 0.02 +"""A slot within this share below the cap counts as clipped.""" +MIN_CAP_W: Final = 200.0 +"""A cap below this is not a cap (night noise, a broken sensor).""" +EXPORT_TOLERANCE_SHARE: Final = 0.03 +EXPORT_TOLERANCE_MIN_W: Final = 50.0 +FORECAST_MARGIN: Final = 0.02 + + +def export_limit_w(site: SiteConfig | None) -> float | None: + """The static feed-in limit in W from the site, or ``None`` (no limit configured).""" + if site is None: + return None + grid = site.grid + if grid.export_limit_w is not None: + return grid.export_limit_w + if grid.export_limit_pct_kwp is not None and site.total_kwp is not None: + return grid.export_limit_pct_kwp / 100 * site.total_kwp * 1000 + return None + + +def plateau_cap_w(pv: TimeSeries, tz: tzinfo) -> float | None: + """The inverter's cap as the data shows it (module doc), or ``None``.""" + known = [v for v in pv.values if v is not None] + if not known: + return None + top = max(known) + if top < MIN_CAP_W: + return None + floor = top * (1 - PLATEAU_TOLERANCE) + days: set[date] = set() + run = 0 + for begin, _, value in pv.slots(): + if value is not None and floor <= value <= top: + run += 1 + if run >= MIN_RUN: + days.add(begin.astimezone(tz).date()) + else: + run = 0 + return top if len(days) >= MIN_CAP_DAYS else None + + +def _value_at(series: TimeSeries | None, begin: datetime) -> float | None: + if series is None or not len(series): + return None + offset = (begin - series.start) / series.step + index = int(offset) + if index != offset or not 0 <= index < len(series): + return None + return series.values[index] + + +def _forecast_higher(reference: TimeSeries | None, begin: datetime, measured: float) -> bool: + """Whether the reference forecast says more than was measured (``True`` if unknown).""" + predicted = _value_at(reference, begin) + if predicted is None: + return reference is None + return predicted > measured * (1 + FORECAST_MARGIN) + + +def clipped_slots( + pv: TimeSeries, + *, + tz: tzinfo, + grid: TimeSeries | None = None, + export_limit: float | None = None, + reference: TimeSeries | None = None, + cap_w: float | None = None, +) -> frozenset[datetime]: + """Start times of the quarter hours of ``pv`` that were limited (module doc). + + ``grid`` and ``reference`` are 15-minute series in W (any extent; slots they lack are + unknown). ``cap_w`` overrides the plateau found in ``pv`` itself. With a ``reference``, + a slot it does not cover is not flagged; without one the forecast condition is skipped. + """ + cap = cap_w if cap_w is not None else plateau_cap_w(pv, tz) + tolerance = ( + None + if export_limit is None + else max(EXPORT_TOLERANCE_SHARE * export_limit, EXPORT_TOLERANCE_MIN_W) + ) + flagged: set[datetime] = set() + for begin, _, measured in pv.slots(): + if measured is None or measured <= 0: + continue + at_cap = cap is not None and measured >= cap * (1 - CAP_BAND) + at_export = False + if export_limit is not None and tolerance is not None and export_limit > 0: + grid_w = _value_at(grid, begin) + at_export = grid_w is not None and -grid_w >= export_limit - tolerance + if (at_cap or at_export) and _forecast_higher(reference, begin, measured): + flagged.add(begin) + return frozenset(flagged) + + +def clip_filter(slots: Iterable[datetime]) -> Callable[[datetime], bool]: + """``is_clipped`` hook over a set of flagged slot starts.""" + flagged = frozenset(slots) + + def is_clipped(slot_start: datetime) -> bool: + return slot_start in flagged + + return is_clipped + + +__all__ = [ + "CAP_BAND", + "MIN_CAP_DAYS", + "MIN_RUN", + "PLATEAU_TOLERANCE", + "clip_filter", + "clipped_slots", + "export_limit_w", + "plateau_cap_w", +] diff --git a/suncore/forecast/pv/factory.py b/suncore/forecast/pv/factory.py new file mode 100644 index 0000000..65b852e --- /dev/null +++ b/suncore/forecast/pv/factory.py @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Build the configured PV forecast sources from the ``forecast`` section (Phase 4 WP5).""" + +from __future__ import annotations + +import structlog + +from suncore.core.config import ForecastConfig +from suncore.forecast.pv.adapter import PvDataReader, TemplateForecastSource +from suncore.templates import ForecastTemplate, TemplateCatalog + +log = structlog.get_logger(__name__) + + +def build_sources( + config: ForecastConfig | None, catalog: TemplateCatalog, reader: PvDataReader +) -> list[TemplateForecastSource]: + """One source per configured forecast template, in the configured order. + + A source whose template is unknown, is not a PV forecast template or is configured without + anything to read is skipped with a warning (the others keep working). + """ + if config is None: + return [] + sources: list[TemplateForecastSource] = [] + for item in config.sources: + template = catalog.get(item.template_id) + if not isinstance(template, ForecastTemplate) or template.forecast_kind != "pv": + log.warning("forecast.pv.template_unusable", template=item.template_id) + continue + entities = {role: ref.entity_id for role, ref in item.entities.items()} + sources.append( + TemplateForecastSource(template, entities, reader, config_entry_id=item.config_entry_id) + ) + return sources + + +__all__ = ["build_sources"] diff --git a/suncore/forecast/pv/learning.py b/suncore/forecast/pv/learning.py new file mode 100644 index 0000000..75f98d8 --- /dev/null +++ b/suncore/forecast/pv/learning.py @@ -0,0 +1,277 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The nightly learning job of the PV forecast (Phase 4 WP5). + +Order, once a night after the time-series maintenance: + +1. **Clipping**: read the measured PV and grid power (15-minute means) and the sources' P50 as + they stood at the start of every local day (their mean is the reference forecast), and flag + the limited quarter hours (:mod:`suncore.forecast.pv.clipping`). +2. **Accuracy** of every source and of the ``blended`` forecast (the one the planner uses) over + the last 14 days, without the clipped quarter hours (:mod:`suncore.forecast.accuracy`). +3. **Bias tables**: every complete local day after a source's ``learned_until`` (at most + ``BACKFILL_DAYS`` back) is learnt from that day-start issue, without the clipped quarter + hours (:mod:`suncore.forecast.pv.calibration`). + +Storage is reached only through protocols (the forecast package never imports storage). A +failing source is logged and keeps last night's state; the job never raises. +""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from datetime import UTC, date, datetime, time, timedelta, tzinfo +from typing import Final, Literal, Protocol + +import structlog + +from suncore.core.forecast import AccuracyRecord +from suncore.core.measurements import SITE_GRID_W, SITE_PV_W +from suncore.core.series import ( + PLANNING_STEP, + Grid, + QuantileSeries, + SeriesUnit, + TimeRange, + TimeSeries, + floor_to_step, + resample_power, +) +from suncore.forecast.accuracy import ( + WINDOW_DAYS, + AccuracyData, + LearnedParams, + update_accuracy, +) +from suncore.forecast.pv.calibration import ( + BiasTable, + learn_day, + load_bias, + new_table, + save_bias, +) +from suncore.forecast.pv.clearsky import PlantGeometry +from suncore.forecast.pv.clipping import clip_filter, clipped_slots + +log = structlog.get_logger(__name__) + +BLENDED_SOURCE_ID: Final = "blended" +"""Source id under which the blended forecast is stored and scored (template ids always +contain a dot, so it cannot collide with a configured source).""" +BACKFILL_DAYS: Final = 60 +"""How far back a source's first learning run (or one after a long pause) reaches.""" + + +class LearningData(AccuracyData, Protocol): + """The part of ``TimeseriesStore`` the nightly job reads.""" + + async def latest_forecast_before( + self, + source_id: str, + kind: Literal["pv", "load", "price"], + before: datetime, + window: TimeRange | None = None, + ) -> QuantileSeries | None: ... + + +@dataclass(frozen=True, slots=True) +class LearningContext: + tz: tzinfo + plant: PlantGeometry | None + export_limit_w: float | None + + +@dataclass(frozen=True, slots=True) +class LocalDay: + day: date + start: datetime + """Local midnight, in UTC.""" + end: datetime + + +@dataclass(frozen=True, slots=True) +class NightlyResult: + accuracy: Mapping[str, Sequence[AccuracyRecord]] + bias: Mapping[str, BiasTable] + clipped_slots: int + + +def local_midnight(moment: datetime, tz: tzinfo) -> datetime: + local = moment.astimezone(tz) + return datetime.combine(local.date(), time(), tzinfo=tz) + + +def local_days(start: datetime, end: datetime, tz: tzinfo) -> list[LocalDay]: + """The complete local days inside ``[start, end)`` (DST days have 23 or 25 hours).""" + first = start.astimezone(tz).date() + if datetime.combine(first, time(), tzinfo=tz) < start: + first += timedelta(days=1) + days: list[LocalDay] = [] + current = first + while True: + # in UTC: Python subtracts two datetimes of the same zone on the wall clock (24 h even + # on a DST day), so the bounds are kept in absolute time + begin = datetime.combine(current, time(), tzinfo=tz).astimezone(UTC) + finish = datetime.combine(current + timedelta(days=1), time(), tzinfo=tz).astimezone(UTC) + if finish > end: + return days + days.append(LocalDay(current, begin, finish)) + current += timedelta(days=1) + + +def slice_series(series: TimeSeries, begin: datetime, end: datetime) -> TimeSeries: + """The grid-aligned part of ``series`` in ``[begin, end)`` (clamped to its extent).""" + step = series.step + first = max(0, (begin - series.start) // step) + last = max(first, min(len(series), -(-(end - series.start) // step))) + return TimeSeries( + start=series.start + step * first, + step=step, + values=series.values[first:last], + unit=series.unit, + ) + + +async def day_start_forecast( + data: LearningData, source_id: str, day: LocalDay +) -> TimeSeries | None: + """The source's P50 for ``day`` as it stood at the day's start, on the 15-minute grid.""" + issue = await data.latest_forecast_before( + source_id, "pv", day.start, TimeRange(start=day.start - timedelta(hours=1), end=day.end) + ) + if issue is None or not len(issue): + return None + grid = Grid(start=day.start, step=PLANNING_STEP, count=(day.end - day.start) // PLANNING_STEP) + series = resample_power(issue.p50, grid) + return series if any(v is not None for v in series.values) else None + + +def mean_reference(window: TimeSeries, forecasts: Sequence[TimeSeries]) -> TimeSeries | None: + """Per slot of ``window`` the mean of the forecasts that know it (``None``: none knows any).""" + sums = [0.0] * len(window) + counts = [0] * len(window) + for forecast in forecasts: + offset = (forecast.start - window.start) // window.step + for i, value in enumerate(forecast.values): + j = offset + i + if value is not None and 0 <= j < len(window): + sums[j] += value + counts[j] += 1 + if not any(counts): + return None + values = tuple(s / c if c else None for s, c in zip(sums, counts, strict=True)) + return TimeSeries(start=window.start, step=window.step, values=values, unit=SeriesUnit.W) + + +async def run_nightly( + data: LearningData, + learned: LearnedParams, + source_ids: Sequence[str], + now: datetime, + context: LearningContext, + *, + window_days: int = WINDOW_DAYS, + backfill_days: int = BACKFILL_DAYS, +) -> NightlyResult: + """Clipping, accuracy (sources + ``blended``) and bias tables; never raises.""" + tz = context.tz + today = local_midnight(now, tz) + tables: dict[str, BiasTable] = {} + try: + tables = await load_bias(learned) + except Exception: + log.exception("forecast.bias.load_failed") + oldest = today - timedelta(days=backfill_days) + learn_from = { + s: max(oldest, tables[s].learned_until or oldest) if s in tables else oldest + for s in source_ids + } + earliest = min([today - timedelta(days=window_days), *learn_from.values()]) + days = local_days(earliest, today, tz) + end = floor_to_step(now, PLANNING_STEP) + clipped: frozenset[datetime] = frozenset() + day_forecasts: dict[tuple[str, date], TimeSeries] = {} + pv: TimeSeries | None = None + if days: + window = TimeRange(start=days[0].start, end=max(end, days[-1].end)) + try: + pv = await data.read_series(SITE_PV_W, window, "15m", SeriesUnit.W) + grid = ( + None + if context.export_limit_w is None + else await data.read_series(SITE_GRID_W, window, "15m", SeriesUnit.W) + ) + for source_id in source_ids: + for day in days: + forecast = await day_start_forecast(data, source_id, day) + if forecast is not None: + day_forecasts[(source_id, day.day)] = forecast + reference = mean_reference(pv, list(day_forecasts.values())) + clipped = clipped_slots( + pv, + tz=tz, + grid=grid, + export_limit=context.export_limit_w, + reference=reference, + ) + except Exception: + log.exception("forecast.clipping.failed") + is_clipped = clip_filter(clipped) + accuracy = await update_accuracy( + data, + learned, + [*source_ids, BLENDED_SOURCE_ID], + now, + window_days=window_days, + is_clipped=is_clipped, + ) + learnt: dict[str, BiasTable] = {} + for source_id in source_ids: + table = tables.get(source_id) or new_table(source_id, context.plant) + if pv is None: + learnt[source_id] = table + continue + try: + for day in days: + if day.start < learn_from[source_id]: + continue + forecast = day_forecasts.get((source_id, day.day)) + actual = slice_series(pv, day.start, day.end) + table = learn_day( + table, + forecast if forecast is not None else slice_series(actual, day.end, day.end), + actual, + plant=context.plant, + tz=tz, + day_end=day.end, + is_clipped=is_clipped, + ) + if table != tables.get(source_id): + await save_bias(learned, table) + except Exception: + log.exception("forecast.bias.failed", source=source_id) + table = tables.get(source_id) or new_table(source_id, context.plant) + learnt[source_id] = table + log.info( + "forecast.nightly_learning", + sources=len(source_ids), + days=len(days), + clipped_slots=len(clipped), + ) + return NightlyResult(accuracy=accuracy, bias=learnt, clipped_slots=len(clipped)) + + +__all__ = [ + "BACKFILL_DAYS", + "BLENDED_SOURCE_ID", + "LearningContext", + "LearningData", + "LocalDay", + "NightlyResult", + "day_start_forecast", + "local_days", + "local_midnight", + "mean_reference", + "run_nightly", + "slice_series", +] diff --git a/suncore/forecast/pv/parsing.py b/suncore/forecast/pv/parsing.py new file mode 100644 index 0000000..a34307f --- /dev/null +++ b/suncore/forecast/pv/parsing.py @@ -0,0 +1,407 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Parsing of PV forecast points into a quantile series in W (Phase 4 WP5; no I/O). + +Every function is pure. The shapes it understands are the ones the research documents +(``docs/research/integrations.md`` "PV forecasts"): + +* a **list of point objects** (Solcast ``detailedForecast``: ``period_start`` ISO 8601 with the + local offset, ``pv_estimate`` / ``pv_estimate10`` / ``pv_estimate90`` in kW average power over + the slot; the action ``query_forecast_data`` answers ``{"data": [same objects]}``); +* a **mapping** ``{ISO timestamp: value}`` (Open-Meteo Solar Forecast ``wh_period_15m``: Wh per + 15 minutes, key = interval start; ``energy/solar_forecast`` ``wh_hours``: Wh per period, key = + period start, UNVERIFIED for Forecast.Solar). + +A point is the value of the period that **starts** at its timestamp. The result lies on the +source's native grid (one regular segment, absolute time, so a DST day with 23 or 25 local +hours needs no special case); a missing period is ``None``, never filled with a neighbour's +value (the one exception is ``missing_is_zero``: Forecast.Solar's integration drops a zero +value that starts exactly at midnight, so a period missing between two known ones is 0). + +Units: power (W, kW) is the average power over the slot and kept; energy (Wh, kWh) per slot is +converted with ``W = Wh * 3600 / slot_seconds``, which preserves the energy exactly. +""" + +from __future__ import annotations + +import itertools +import math +from collections import Counter +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from datetime import datetime, timedelta +from functools import reduce +from typing import Any, Final + +from suncore.core.series import ( + QuantileSeries, + SeriesUnit, + TimeSeries, + floor_to_step, + segments_from_points, +) +from suncore.core.units import Dimension, Unit, to_base +from suncore.templates.models import SeriesFields + +NEGATIVE_TOLERANCE: Final = 1e-6 +"""A forecast below ``-tolerance`` (in the source's unit) is invalid data, not a small error.""" + + +@dataclass(frozen=True, slots=True) +class RawPoint: + """One parsed point in the template's unit; ``None`` = the value is missing or invalid.""" + + start: datetime + value: float | None + p10: float | None = None + p90: float | None = None + + +@dataclass(frozen=True, slots=True) +class ParsedPoints: + points: tuple[RawPoint, ...] + skipped: int + """Entries that could not be used at all (no parsable timestamp, not an object).""" + + +def parse_timestamp(raw: object) -> datetime | None: + """An ISO 8601 timestamp **with** a UTC offset; anything else is ``None``. + + A timestamp without an offset is refused: the documented sources always carry one, and + guessing a zone is how forecasts end up an hour off on a DST day. + """ + if isinstance(raw, datetime): + moment = raw + elif isinstance(raw, str): + try: + moment = datetime.fromisoformat(raw.strip()) + except ValueError: + return None + else: + return None + return moment if moment.tzinfo is not None and moment.utcoffset() is not None else None + + +def number(raw: object) -> float | None: + """A finite number (not a bool); numeric strings are accepted, everything else is ``None``.""" + if isinstance(raw, bool): + return None + if isinstance(raw, int | float): + value = float(raw) + elif isinstance(raw, str): + try: + value = float(raw.strip()) + except ValueError: + return None + else: + return None + return value if math.isfinite(value) else None + + +def dig(raw: object, path: str | None) -> object: + """Follow a dotted path of mapping keys (``None`` path = ``raw`` itself).""" + current = raw + if path is None: + return current + for key in path.split("."): + if not isinstance(current, Mapping) or key not in current: + return None + current = current[key] + return current + + +def dig_response(raw: object, path: str) -> list[Any]: + """Entries at ``path`` of an action response; a ``*`` segment spreads over a mapping's values. + + ``data`` -> the list at ``data``; ``prices.*`` -> the lists of every value of ``prices`` + concatenated. A path that does not lead to lists gives an empty list. + """ + nodes: list[object] = [raw] + for segment in path.split("."): + nxt: list[object] = [] + for node in nodes: + if not isinstance(node, Mapping): + continue + if segment == "*": + nxt.extend(node.values()) + elif segment in node: + nxt.append(node[segment]) + nodes = nxt + out: list[Any] = [] + for node in nodes: + if isinstance(node, list | tuple): + out.extend(node) + return out + + +def list_points(raw: object, fields: SeriesFields) -> ParsedPoints: + """Points from a list of objects named by ``fields`` (``None`` or a non-list: no points).""" + if not isinstance(raw, list | tuple): + return ParsedPoints((), 0) + ts_field, value_field = fields.timestamp_field, fields.value_field + if ts_field is None or value_field is None: + msg = "list_points needs timestamp_field and value_field" + raise ValueError(msg) + points: list[RawPoint] = [] + skipped = 0 + for entry in raw: + if not isinstance(entry, Mapping): + skipped += 1 + continue + start = parse_timestamp(entry.get(ts_field)) + if start is None: + skipped += 1 + continue + value = number(entry.get(value_field)) + p10 = number(entry.get(fields.p10_field)) if fields.p10_field is not None else None + p90 = number(entry.get(fields.p90_field)) if fields.p90_field is not None else None + points.append(RawPoint(start, value, p10, p90)) + return ParsedPoints(tuple(points), skipped) + + +def mapping_points(raw: object) -> ParsedPoints: + """Points from ``{ISO timestamp: value}`` (``None`` or a non-mapping: no points).""" + if not isinstance(raw, Mapping): + return ParsedPoints((), 0) + points: list[RawPoint] = [] + skipped = 0 + for key, value in raw.items(): + start = parse_timestamp(key) + if start is None: + skipped += 1 + continue + points.append(RawPoint(start, number(value))) + return ParsedPoints(tuple(points), skipped) + + +def infer_step(points: Sequence[RawPoint], default: timedelta) -> timedelta: + """The period length from the spacing of the timestamps (their greatest common divisor). + + ``energy/solar_forecast`` has no unit or period field (the type is only "Wh per timestamp"), + so the length is inferred and never assumed; one point or none: ``default``. + """ + stamps = sorted({p.start for p in points}) + gaps = {int((b - a).total_seconds()) for a, b in itertools.pairwise(stamps)} + gaps.discard(0) + if not gaps: + return default + divisor = reduce(math.gcd, gaps) + return timedelta(seconds=max(divisor, 60)) + + +ALLOWED_STEPS: Final = (timedelta(minutes=60), timedelta(minutes=30), timedelta(minutes=15)) +"""Period lengths an energy mapping may have (``energy/solar_forecast``).""" + + +@dataclass(frozen=True, slots=True) +class Regularized: + """Energy points brought onto one grid (:func:`regularize_energy`).""" + + points: tuple[RawPoint, ...] + step: timedelta + dropped: int + """Off-grid points without energy (0 Wh or unknown) that were left out.""" + merged: int + """Off-grid points whose energy was added to the period that encloses them.""" + + +def _on_grid(moment: datetime, step: timedelta) -> bool: + return floor_to_step(moment, step) == moment + + +def _sum(a: float | None, b: float | None) -> float | None: + return None if a is None or b is None else a + b + + +def regularize_energy(points: Sequence[RawPoint], default: timedelta) -> Regularized: + """Energy points (Wh per period, key = period start) on one grid of 15, 30 or 60 minutes. + + The period is the most frequent gap between consecutive points that lie on the 15-minute + grid, if that gap is 15, 30 or 60 minutes (a tie takes the longer one; no such gap: + ``default``). A point that is not on the grid of that period - a sunrise or sunset key such + as ``07:41:13`` that a vendor may emit (UNVERIFIED for Forecast.Solar) - never fails the + source: without energy (0 Wh or unknown) it is dropped, with energy it is **added to the + period that encloses it** (energy preserving; P10/P90 only where both points have them). + """ + base = ALLOWED_STEPS[-1] + aligned = sorted({p.start for p in points if _on_grid(p.start, base)}) + gaps = Counter(b - a for a, b in itertools.pairwise(aligned) if b - a in ALLOWED_STEPS) + step = default + if gaps: + top = max(gaps.values()) + step = max(g for g, count in gaps.items() if count == top) + regular: dict[datetime, RawPoint] = {} + off: list[RawPoint] = [] + for point in points: + if _on_grid(point.start, step): + regular[point.start] = point # the later one wins, like _dedupe + else: + off.append(point) + dropped = merged = 0 + for point in off: + if point.value is None or abs(point.value) <= NEGATIVE_TOLERANCE: + dropped += 1 + continue + key = floor_to_step(point.start, step) + current = regular.get(key) + if current is None: + regular[key] = RawPoint(key, point.value, point.p10, point.p90) + elif current.value is None: + regular[key] = RawPoint(current.start, point.value, point.p10, point.p90) + else: + regular[key] = RawPoint( + current.start, + current.value + point.value, + _sum(current.p10, point.p10), + _sum(current.p90, point.p90), + ) + merged += 1 + ordered = tuple(sorted(regular.values(), key=lambda p: p.start)) + return Regularized(ordered, step, dropped, merged) + + +def _dedupe(points: Sequence[RawPoint]) -> list[RawPoint]: + """Sorted by time, one point per instant (the later one in input order wins).""" + unique = {p.start.timestamp(): p for p in points} + return [unique[key] for key in sorted(unique)] + + +def _fill_missing_as_zero( + points: list[RawPoint], step: timedelta, *, quantiles: bool +) -> list[RawPoint]: + out: list[RawPoint] = [] + for point in points: + if out: + previous = out[-1].start + gap = point.start - previous + if gap > step and gap % step == timedelta(0): + out.extend( + RawPoint( + previous + step * (i + 1), + 0.0, + 0.0 if quantiles else None, + 0.0 if quantiles else None, + ) + for i in range(gap // step - 1) + ) + out.append(point) + return out + + +def _watts(value: float | None, unit: Unit, step: timedelta) -> float | None: + if value is None or value < -NEGATIVE_TOLERANCE: + return None + base = max(to_base(value, unit), 0.0) + if unit.dimension is Dimension.ENERGY: + return base * 3600.0 / step.total_seconds() + return base + + +def _segment( + stamps: Sequence[datetime], values: Sequence[float | None], step: timedelta +) -> TimeSeries: + segments = segments_from_points( + list(zip(stamps, values, strict=True)), SeriesUnit.W, expected_step=step + ) + if len(segments) != 1: # only one accepted step: a single regular segment + msg = "forecast points do not form one regular series" + raise ValueError(msg) + return segments[0] + + +def build_quantiles( + points: Sequence[RawPoint], + unit: Unit, + step: timedelta, + *, + missing_is_zero: bool = False, +) -> QuantileSeries | None: + """The points as P10/P50/P90 in W on the grid of ``step``; ``None`` if there are none. + + * the value of a point without a known P50 is unknown in all three quantiles; + * P10/P90 that a point lacks stay ``None`` (an interval is not invented), and a source + without quantile fields gets equal quantiles (``QuantileSeries.deterministic``); + * an inverted pair (P10 > P50 or P90 < P50) is clamped to P50 - the safe direction; + * a negative value is invalid data and becomes ``None``. + + Raises: + ValueError: timestamps that are not on the grid of ``step`` (an irregular source is + refused as a whole, SunCore never guesses). + """ + ordered = _dedupe(points) + if not ordered: + return None + has_quantiles = any(p.p10 is not None or p.p90 is not None for p in ordered) + if missing_is_zero: + ordered = _fill_missing_as_zero(ordered, step, quantiles=has_quantiles) + p10s: list[float | None] = [] + p50s: list[float | None] = [] + p90s: list[float | None] = [] + for point in ordered: + p50 = _watts(point.value, unit, step) + p10 = _watts(point.p10, unit, step) if p50 is not None else None + p90 = _watts(point.p90, unit, step) if p50 is not None else None + if p50 is not None: + if p10 is not None: + p10 = min(p10, p50) + if p90 is not None: + p90 = max(p90, p50) + p50s.append(p50) + p10s.append(p10) + p90s.append(p90) + stamps = [p.start for p in ordered] + series50 = _segment(stamps, p50s, step) + if not has_quantiles: + return QuantileSeries.deterministic(series50) + return QuantileSeries( + p10=_segment(stamps, p10s, step), p50=series50, p90=_segment(stamps, p90s, step) + ) + + +def has_known(series: QuantileSeries) -> bool: + """Whether any P50 value is known.""" + return any(v is not None for v in series.p50.values) + + +def crop(series: QuantileSeries, start: datetime, end: datetime) -> QuantileSeries: + """The slots of ``series`` that overlap ``[start, end)``; never padded.""" + step = series.step + first = max(0, math.floor((start - series.start) / step)) + last = min(len(series), math.ceil((end - series.start) / step)) + last = max(first, last) + + def cut(part: TimeSeries) -> TimeSeries: + return TimeSeries( + start=part.start + step * first, + step=step, + values=part.values[first:last], + unit=part.unit, + ) + + return QuantileSeries(p10=cut(series.p10), p50=cut(series.p50), p90=cut(series.p90)) + + +def empty_series(start: datetime, step: timedelta) -> QuantileSeries: + """A series without slots (the answer of a source that has nothing).""" + empty = TimeSeries(start=start, step=step, values=(), unit=SeriesUnit.W) + return QuantileSeries(p10=empty, p50=empty, p90=empty) + + +__all__ = [ + "ALLOWED_STEPS", + "NEGATIVE_TOLERANCE", + "ParsedPoints", + "RawPoint", + "Regularized", + "build_quantiles", + "crop", + "dig", + "dig_response", + "empty_series", + "has_known", + "infer_step", + "list_points", + "mapping_points", + "number", + "parse_timestamp", + "regularize_energy", +] diff --git a/suncore/forecast/pv/service.py b/suncore/forecast/pv/service.py new file mode 100644 index 0000000..01a071e --- /dev/null +++ b/suncore/forecast/pv/service.py @@ -0,0 +1,626 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The PV forecast service: refresh, keep the last good data, store, expose (Phase 4 WP5). + +* **Refresh** every 30 minutes and whenever one of the sources' entities changes (the runtime + calls :meth:`PvForecastService.on_entities`; at most one refresh per ``min_refresh_gap_s``). +* **Keep the last good data.** A fetch that yields nothing (an unavailable sensor, an empty or + all-``None`` series, data that lies entirely in the past while the old data still reaches into + the future, an exception) never replaces what is held; only the status changes. Slots a fetch + lacks are taken from the previous good fetch for up to ``gap_fill_max_age`` after the last + *complete* fetch (``filled_slots``), so one flapping day sensor does not cost the plan its + tomorrow. +* **Store** the fetched forecast in ``timeseries.db`` (kind ``pv``, the next ``horizon`` from the + current slot) at most once an hour, or earlier - but not within ``min_store_gap_s`` - when it + changed materially against the last stored issue (decision from the review of PR 4-B: a forecast + refresh every few minutes would write 18 000 rows a day per source). What is stored is what + the source said, never the gap-filled series. +* **Expose** per source the latest series with its freshness, and the *blended* forecast the + planner uses: every source on the 15-minute planning grid from now to +48 h, bias-calibrated + where its calibration is active (:mod:`~suncore.forecast.pv.calibration`), then blended by + measured skill (:mod:`~suncore.forecast.pv.blend`). The blend is stored like a source (id + ``blended``) so the accuracy job scores what the planner actually got. +* **Learning state** (accuracy records, bias tables) is handed in by the runtime after the + nightly job (:meth:`PvForecastService.set_learning`); the location and time zone come through + the ``context`` callable because they become known only after Home Assistant answers. + +Time and sleeping are injected; the service never reads the wall clock. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence +from dataclasses import dataclass, field +from datetime import UTC, datetime, timedelta, tzinfo +from typing import Final, Literal, Protocol + +import structlog + +from suncore.core.forecast import AccuracyRecord, ForecastSource +from suncore.core.series import ( + PLANNING_STEP, + QuantileSeries, + TimeRange, + TimeSeries, + floor_to_step, +) +from suncore.forecast.pv.adapter import FetchReport, FetchStatus +from suncore.forecast.pv.blend import BlendInput, BlendResult, blend +from suncore.forecast.pv.calibration import ( + BiasTable, + CalibrationState, + calibrate, + calibration_state, +) +from suncore.forecast.pv.clearsky import PlantGeometry +from suncore.forecast.pv.learning import BLENDED_SOURCE_ID +from suncore.forecast.pv.parsing import crop, has_known +from suncore.forecast.views import ( + MAX_VIEW_HOURS, + BlendedView, + BlendWeightsView, + CalibrationView, + PvForecastView, + PvSeriesView, + PvSourceView, +) + +log = structlog.get_logger(__name__) + +REFRESH_INTERVAL_S: Final = 1800.0 +STORE_INTERVAL_S: Final = 3600.0 +MIN_STORE_GAP_S: Final = 900.0 +MIN_REFRESH_GAP_S: Final = 30.0 +HORIZON: Final = timedelta(hours=48) +GAP_FILL_MAX_AGE: Final = timedelta(hours=6) +MATERIAL_CHANGE: Final = 0.15 +"""Relative L1 difference of the next 24 h that makes a forecast worth an extra stored issue.""" +MATERIAL_REFERENCE_WH: Final = 1000.0 +"""Floor of the denominator, so a night-time forecast of a few Wh never counts as a change.""" +STALE_AFTER_REFRESHES: Final = 3 +DEFAULT_MIN_HISTORY_DAYS: Final = 30 +BLEND_STALE_AFTER: Final = timedelta(hours=12) +"""A source whose last good fetch is older is left out of the blend while a fresher one exists.""" + + +@dataclass(frozen=True, slots=True) +class SourceSettings: + """How a source is weighted (``forecast.sources[].weight_mode`` / ``weight``).""" + + weight_mode: Literal["auto", "manual"] = "auto" + manual_weight: float | None = None + + +@dataclass(frozen=True, slots=True) +class BlendContext: + """What the blend needs from the site: time zone and (if known) the plant's geometry.""" + + tz: tzinfo = UTC + plant: PlantGeometry | None = None + + +@dataclass(frozen=True, slots=True) +class BlendedForecast: + """The forecast the planner uses, with how it was made.""" + + series: QuantileSeries + """Calibrated where active, blended; 15-minute grid from the current slot.""" + result: BlendResult + uncalibrated: QuantileSeries + """The blend of the sources as they said it.""" + calibrated: QuantileSeries | None + """The blend with every learnt calibration applied, active or not (``None``: no source has + a bias table yet) - lets the user compare before the calibration switches on.""" + calibration: tuple[CalibrationState, ...] + stale: bool = False + """Every source with data is older than ``BLEND_STALE_AFTER`` (they are used anyway).""" + excluded: tuple[str, ...] = () + """Sources left out because they are stale while a fresher one exists.""" + + @property + def calibration_active(self) -> bool: + return any(c.active for c in self.calibration) + + +class ReportingSource(ForecastSource, Protocol): + """A forecast source that tells what it found and which entities it watches.""" + + @property + def last_report(self) -> FetchReport: ... + + @property + def watched_entities(self) -> frozenset[str]: ... + + +class ForecastStore(Protocol): + """The part of ``TimeseriesStore`` the service writes to.""" + + async def store_forecast( + self, + source_id: str, + kind: Literal["pv", "load", "price"], + issued_at: datetime, + forecast: QuantileSeries | TimeSeries, + ) -> int: ... + + +@dataclass(slots=True) +class SourceSnapshot: + """What the service knows about one source.""" + + source_id: str + series: QuantileSeries | None = None + fetched_at: datetime | None = None + complete_at: datetime | None = None + """Last fetch that needed no gap filling (gap filling is allowed for a while after it).""" + attempted_at: datetime | None = None + status: str = FetchStatus.NO_DATA.value + report: FetchReport | None = None + filled_slots: int = 0 + error: str | None = None + last_stored_at: datetime | None = None + last_stored: QuantileSeries | None = field(default=None, repr=False) + + def age_s(self, now: datetime) -> float | None: + return None if self.fetched_at is None else (now - self.fetched_at).total_seconds() + + +def materially_changed(old: QuantileSeries, new: QuantileSeries, now: datetime) -> bool: + """Whether the next 24 h of ``new`` differ from ``old`` by more than ``MATERIAL_CHANGE``. + + Compares the P50 of the slots both know, as the sum of absolute energy differences relative to + the energy of the older issue (at least ``MATERIAL_REFERENCE_WH``). + """ + horizon_end = now + timedelta(hours=24) + a = crop(old, now, horizon_end) + b = crop(new, now, horizon_end) + if a.step != b.step: + return True + diff = 0.0 + base = 0.0 + hours = a.step / timedelta(hours=1) + for begin, _, old_value in a.p50.slots(): + offset = (begin - b.start) / b.step + if offset != int(offset) or not 0 <= int(offset) < len(b): + continue + new_value = b.p50.values[int(offset)] + if old_value is None or new_value is None: + continue + diff += abs(new_value - old_value) * hours + base += old_value * hours + return diff > MATERIAL_CHANGE * max(base, MATERIAL_REFERENCE_WH) + + +def fill_from_previous( + new: QuantileSeries, previous: QuantileSeries, now: datetime +) -> tuple[QuantileSeries, int]: + """``new`` with its unknown future slots taken from ``previous`` (same grid only). + + Returns the series and the number of slots filled. A slot that ended before ``now`` is not + filled. The result may extend ``new`` to where ``previous`` reaches. + """ + step = new.step + if previous.step != step or (new.start - previous.start) % step != timedelta(0): + return new, 0 + shift = (new.start - previous.start) // step # index in previous of new's slot 0 + last_prev = len(previous) - shift # new-index just past previous's end + total = max(len(new), last_prev) + quantiles = ([*new.p10.values], [*new.p50.values], [*new.p90.values]) + for column in quantiles: + column.extend([None] * (total - len(column))) + filled = 0 + longest = len(new) + for j in range(total): + slot_end = new.start + step * (j + 1) + k = j + shift + if ( + quantiles[1][j] is not None + or slot_end <= now + or not 0 <= k < len(previous) + or previous.p50.values[k] is None + ): + continue + quantiles[0][j] = previous.p10.values[k] + quantiles[1][j] = previous.p50.values[k] + quantiles[2][j] = previous.p90.values[k] + filled += 1 + longest = max(longest, j + 1) + if filled == 0: + return new, 0 + + def part(values: list[float | None]) -> TimeSeries: + return TimeSeries(start=new.start, step=step, values=tuple(values[:longest]), unit=new.unit) + + return QuantileSeries( + p10=part(quantiles[0]), p50=part(quantiles[1]), p90=part(quantiles[2]) + ), filled + + +def _future_known(series: QuantileSeries, now: datetime) -> bool: + return any(end > now and v is not None for _, end, v in series.p50.slots()) + + +class PvForecastService: + """Refreshes the sources, keeps their last good series and stores them.""" + + def __init__( + self, + sources: Sequence[ReportingSource], + *, + clock: Callable[[], datetime], + sleep: Callable[[float], Awaitable[None]] = asyncio.sleep, + store: ForecastStore | None = None, + refresh_interval_s: float = REFRESH_INTERVAL_S, + store_interval_s: float = STORE_INTERVAL_S, + min_store_gap_s: float = MIN_STORE_GAP_S, + min_refresh_gap_s: float = MIN_REFRESH_GAP_S, + horizon: timedelta = HORIZON, + gap_fill_max_age: timedelta = GAP_FILL_MAX_AGE, + settings: Mapping[str, SourceSettings] | None = None, + min_history_days: int = DEFAULT_MIN_HISTORY_DAYS, + context: Callable[[], BlendContext] = BlendContext, + ) -> None: + self._sources = list(sources) + self._settings = dict(settings or {}) + self._min_history_days = min_history_days + self._context = context + self._accuracy: dict[str, tuple[AccuracyRecord, ...]] = {} + self._bias: dict[str, BiasTable] = {} + self._blend_state = SourceSnapshot(BLENDED_SOURCE_ID) + self._clock = clock + self._sleep = sleep + self._store = store + self._refresh_interval_s = refresh_interval_s + self._store_interval = timedelta(seconds=store_interval_s) + self._min_store_gap = timedelta(seconds=min_store_gap_s) + self._min_refresh_gap_s = min_refresh_gap_s + self._horizon = horizon + self._gap_fill_max_age = gap_fill_max_age + self._state = {s.source_id: SourceSnapshot(s.source_id) for s in self._sources} + self._watched = frozenset().union(*(s.watched_entities for s in self._sources)) + self._woken = False + self._sleeper: asyncio.Future[None] | None = None + self._lock = asyncio.Lock() + self._last_refresh_at: datetime | None = None + + # --- inputs --------------------------------------------------------------------- + + @property + def source_ids(self) -> list[str]: + return [s.source_id for s in self._sources] + + @property + def watched_entities(self) -> frozenset[str]: + return self._watched + + def on_entities(self, entity_ids: Iterable[str]) -> None: + """Request a refresh if one of the sources' entities changed (cheap, sync).""" + if not self._watched.isdisjoint(entity_ids): + self.wake() + + def wake(self) -> None: + """Request a refresh soon (debounced like an entity change), e.g. after Home Assistant + (re)connected: ``energy/solar_forecast`` sources watch no entity and would otherwise + wait for the next interval.""" + self._woken = True + if self._sleeper is not None: + self._sleeper.cancel() + + # --- refresh -------------------------------------------------------------------- + + async def refresh(self) -> list[SourceSnapshot]: + """Fetch every source once; failures are logged and never raised.""" + async with self._lock: + now = self._clock() + self._last_refresh_at = now + begin = floor_to_step(now, PLANNING_STEP) + window = TimeRange(start=begin, end=begin + self._horizon) + for source in self._sources: + await self._refresh_one(source, window, now) + await self._store_blend(now) + return list(self._state.values()) + + async def _store_blend(self, now: datetime) -> None: + try: + blended = self.blended(now) + except Exception: + log.exception("forecast.pv.blend_failed") + return + if blended is not None and has_known(blended.series): + await self._maybe_store(BLENDED_SOURCE_ID, self._blend_state, blended.series, now) + + async def _refresh_one(self, source: ReportingSource, window: TimeRange, now: datetime) -> None: + state = self._state[source.source_id] + state.attempted_at = now + try: + fetched = await source.fetch(window) + except asyncio.CancelledError: + raise + except Exception as exc: + log.warning("forecast.pv.fetch_failed", source=source.source_id, error=repr(exc)) + state.status = FetchStatus.ERROR.value + state.error = type(exc).__name__ + return + report = source.last_report + state.report = report + state.error = None + if not has_known(fetched): + state.status = report.status.value + return + if ( + state.series is not None + and _future_known(state.series, now) + and not _future_known(fetched, now) + ): + log.warning("forecast.pv.fetch_in_the_past", source=source.source_id) + state.status = FetchStatus.NO_DATA.value + return + effective, filled = fetched, 0 + if ( + state.series is not None + and state.complete_at is not None + and now - state.complete_at <= self._gap_fill_max_age + ): + effective, filled = fill_from_previous(fetched, state.series, now) + state.series = effective + state.filled_slots = filled + state.fetched_at = now + state.status = report.status.value + if filled == 0: + state.complete_at = now + await self._maybe_store(source.source_id, state, fetched, now) + + async def _maybe_store( + self, source_id: str, state: SourceSnapshot, fetched: QuantileSeries, now: datetime + ) -> None: + if self._store is None or not self._store_due(state, fetched, now): + return + begin = floor_to_step(now, PLANNING_STEP) + issue = crop(fetched, begin, begin + self._horizon) + if not has_known(issue): + return + try: + await self._store.store_forecast(source_id, "pv", now, issue) + except asyncio.CancelledError: + raise + except Exception: + log.exception("forecast.pv.store_failed", source=source_id) + return + state.last_stored_at = now + state.last_stored = issue + + def _store_due(self, state: SourceSnapshot, fetched: QuantileSeries, now: datetime) -> bool: + if state.last_stored_at is None or state.last_stored is None: + return True + age = now - state.last_stored_at + if age >= self._store_interval: + return True + return age >= self._min_store_gap and materially_changed(state.last_stored, fetched, now) + + # --- loop ----------------------------------------------------------------------- + + async def run(self) -> None: + """Refresh now, then every interval or when a watched entity changes.""" + while True: + await self.refresh() + await self._wait() + + async def _wait(self) -> None: + """Sleep for the refresh interval; an entity change ends the sleep early (debounced). + + A change that arrived while the refresh ran is not lost: it skips the sleep. + """ + if not self._woken: + self._sleeper = asyncio.ensure_future(self._sleep(self._refresh_interval_s)) + try: + await self._sleeper + except asyncio.CancelledError: + task = asyncio.current_task() + if task is not None and task.cancelling(): + raise # the service itself is being stopped, not woken + finally: + self._sleeper = None + woke, self._woken = self._woken, False + if woke: + await self._sleep(self._min_refresh_gap_s) # debounce a burst of entity updates + + # --- outputs -------------------------------------------------------------------- + + def snapshot(self, source_id: str) -> SourceSnapshot | None: + return self._state.get(source_id) + + def snapshots(self) -> list[SourceSnapshot]: + return [self._state[s.source_id] for s in self._sources] + + def set_learning( + self, + accuracy: Mapping[str, Sequence[AccuracyRecord]], + bias: Mapping[str, BiasTable], + ) -> None: + """Hand in the nightly job's results (accuracy records per source, bias tables).""" + self._accuracy = {k: tuple(v) for k, v in accuracy.items()} + self._bias = dict(bias) + + def calibration_states(self) -> list[CalibrationState]: + plant = self._context().plant + return [ + calibration_state( + s.source_id, + self._bias.get(s.source_id), + plant=plant, + min_history_days=self._min_history_days, + ) + for s in self._sources + ] + + def blended(self, now: datetime | None = None) -> BlendedForecast | None: + """The forecast the planner uses: every source with data reaching into the future, + calibrated where its calibration is active, blended on the planning grid from the + current slot for ``horizon``; ``None`` when no source has data.""" + moment = self._clock() if now is None else now + context = self._context() + states = {c.source_id: c for c in self.calibration_states()} + raw: list[BlendInput] = [] + applied: list[BlendInput] = [] + learnt: list[BlendInput] = [] + usable = [ + snap + for snap in self.snapshots() + if snap.series is not None and _future_known(snap.series, moment) + ] + fresh = [ + snap + for snap in usable + if snap.fetched_at is not None and moment - snap.fetched_at <= BLEND_STALE_AFTER + ] + chosen = fresh or usable + chosen_ids = {s.source_id for s in chosen} + excluded = tuple(s.source_id for s in usable if s.source_id not in chosen_ids) + for snap in chosen: + if snap.series is None: + continue + setting = self._settings.get(snap.source_id, SourceSettings()) + item = BlendInput( + source_id=snap.source_id, + series=snap.series, + weight_mode=setting.weight_mode, + manual_weight=setting.manual_weight, + records=self._accuracy.get(snap.source_id, ()), + ) + raw.append(item) + table = self._bias.get(snap.source_id) + if table is None or not table.cells: + applied.append(item) + learnt.append(item) + continue + fixed = _replace_series( + item, calibrate(snap.series, table, plant=context.plant, tz=context.tz) + ) + learnt.append(fixed) + applied.append(fixed if states[snap.source_id].active else item) + result = blend(applied, moment, horizon=self._horizon) + uncalibrated = blend(raw, moment, horizon=self._horizon) + if result is None or uncalibrated is None: + return None + has_table = any(a is not b for a, b in zip(raw, learnt, strict=True)) + calibrated = blend(learnt, moment, horizon=self._horizon) if has_table else None + return BlendedForecast( + series=result.series, + result=result, + uncalibrated=uncalibrated.series, + calibrated=None if calibrated is None else calibrated.series, + calibration=tuple(states.values()), + stale=bool(usable) and not fresh, + excluded=excluded, + ) + + def is_stale(self, snap: SourceSnapshot, now: datetime) -> bool: + age = snap.age_s(now) + return age is None or age > STALE_AFTER_REFRESHES * self._refresh_interval_s + + def view(self, hours: int = 48) -> PvForecastView: + """The API read model: every source cropped to ``hours`` from the current slot.""" + now = self._clock() + hours = min(max(hours, 1), MAX_VIEW_HOURS) + begin = floor_to_step(now, PLANNING_STEP) + end = begin + timedelta(hours=hours) + sources = [self._source_view(snap, begin, end, now) for snap in self.snapshots()] + chosen = self.blended(now) + blended = None if chosen is None else _blended_view(chosen, begin, end) + return PvForecastView(generated_at=now, hours=hours, sources=sources, blended=blended) + + def _source_view( + self, snap: SourceSnapshot, begin: datetime, end: datetime, now: datetime + ) -> PvSourceView: + series = None + if snap.series is not None: + cut = crop(snap.series, begin, end) + series = PvSeriesView( + start=cut.start, + step_s=int(cut.step.total_seconds()), + p10_w=list(cut.p10.values), + p50_w=list(cut.p50.values), + p90_w=list(cut.p90.values), + ) + return PvSourceView( + source_id=snap.source_id, + status=FetchStatus(snap.status), + fetched_at=snap.fetched_at, + age_s=snap.age_s(now), + stale=self.is_stale(snap, now), + filled_slots=snap.filled_slots, + series=series, + ) + + +def _replace_series(item: BlendInput, series: QuantileSeries) -> BlendInput: + return BlendInput( + source_id=item.source_id, + series=series, + weight_mode=item.weight_mode, + manual_weight=item.manual_weight, + records=item.records, + ) + + +def _blended_view(chosen: BlendedForecast, begin: datetime, end: datetime) -> BlendedView: + cut = crop(chosen.series, begin, end) + raw = crop(chosen.uncalibrated, begin, end) + learnt = None if chosen.calibrated is None else crop(chosen.calibrated, begin, end) + return BlendedView( + method=chosen.result.method, + sources=list(chosen.result.sources), + start=cut.start, + step_s=int(cut.step.total_seconds()), + p10_w=list(cut.p10.values), + p50_w=list(cut.p50.values), + p90_w=list(cut.p90.values), + uncalibrated_p50_w=list(raw.p50.values), + calibrated_p50_w=None if learnt is None else list(learnt.p50.values), + calibration_active=chosen.calibration_active, + stale=chosen.stale, + excluded_sources=list(chosen.excluded), + weights=[ + BlendWeightsView( + horizon=w.horizon, + basis=w.auto_basis, + manual=w.manual, + weights=dict(w.weights), + ) + for w in chosen.result.weights + ], + calibration=[ + CalibrationView( + source_id=c.source_id, + active=c.active, + reason=c.reason, + days_with_data=c.days_with_data, + min_history_days=c.min_history_days, + cells=c.cells, + factor_min=c.factor_min, + factor_max=c.factor_max, + ) + for c in chosen.calibration + ], + ) + + +__all__ = [ + "BLEND_STALE_AFTER", + "DEFAULT_MIN_HISTORY_DAYS", + "GAP_FILL_MAX_AGE", + "HORIZON", + "MATERIAL_CHANGE", + "MIN_STORE_GAP_S", + "REFRESH_INTERVAL_S", + "STORE_INTERVAL_S", + "BlendContext", + "BlendedForecast", + "ForecastStore", + "PvForecastService", + "ReportingSource", + "SourceSettings", + "SourceSnapshot", + "fill_from_previous", + "materially_changed", +] diff --git a/suncore/forecast/views.py b/suncore/forecast/views.py new file mode 100644 index 0000000..1bdffa7 --- /dev/null +++ b/suncore/forecast/views.py @@ -0,0 +1,154 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Read models of the PV forecast and its accuracy (what the API serves; Phase 4 WP5). + +Values are W; ``None`` is an unknown slot (never zero). ``status`` is a +:class:`~suncore.forecast.pv.adapter.FetchStatus` value, rendered by the UI as +``forecast.status.``. +""" + +from __future__ import annotations + +from typing import Annotated, Literal + +from pydantic import AwareDatetime, BaseModel, ConfigDict, Field + +from suncore.core.forecast import AccuracyRecord, HorizonBucket +from suncore.forecast.pv.adapter import FetchStatus + +MAX_VIEW_HOURS = 96 +"""Largest ``hours`` the API serves.""" + + +class _View(BaseModel): + model_config = ConfigDict(frozen=True, extra="forbid") + + +class PvSeriesView(_View): + """P10/P50/P90 in W on one regular grid: ``p50_w[i]`` covers ``[start + i*step_s, ...)``.""" + + start: AwareDatetime + step_s: Annotated[int, Field(gt=0)] + p10_w: list[float | None] + p50_w: list[float | None] + p90_w: list[float | None] + + +class PvSourceView(_View): + """One source: its latest good series and how fresh it is.""" + + source_id: str + status: FetchStatus + """Result of the latest attempt (``ok``, ``partial``, ``no_data``, ``error``, ...).""" + fetched_at: AwareDatetime | None + """When the source last delivered data; ``None`` = never.""" + age_s: float | None + stale: bool + """No data yet, or older than three refresh intervals.""" + filled_slots: int + """Slots taken from the previous good fetch because the latest one lacked them.""" + series: PvSeriesView | None + + +class BlendWeightsView(_View): + """The weights of one horizon bucket (before the per-slot renormalisation over the sources + that have a value in a slot).""" + + horizon: HorizonBucket + basis: Literal["inverse_mae", "equal_no_history"] | None + """How the auto-weighted sources were weighted: by the inverse of their 14-day MAE, or + equally because a source lacks enough accuracy history; ``None`` = all weights manual.""" + manual: bool + """A manually weighted source took part.""" + weights: dict[str, float] + """Source id to weight; the weights sum to 1.""" + + +class CalibrationView(_View): + """The bias calibration of one source. + + ``reason``: ``active`` (applied), ``collecting`` (learnt and stored but not applied before + ``min_history_days`` days with data), ``no_history`` (nothing learnt yet), + ``classifier_changed`` (learnt with the other weather classifier; waits). + """ + + source_id: str + active: bool + reason: Literal["active", "collecting", "no_history", "classifier_changed"] + days_with_data: int + min_history_days: int + cells: int + """Cells (hour x season x weather class) with samples.""" + factor_min: float | None + factor_max: float | None + + +class BlendedView(_View): + """The forecast the planner uses: the sources on the 15-minute grid, bias-calibrated where + the calibration is active, blended by measured skill (P10/P50/P90 in W). + + ``uncalibrated_p50_w`` is the blend of the sources as they said it; ``calibrated_p50_w`` the + blend with every learnt calibration applied, active or not (``None`` while no source has + learnt anything), so the effect can be compared before it switches on. + """ + + method: Literal["weighted", "single_source"] + sources: list[str] + start: AwareDatetime + step_s: Annotated[int, Field(gt=0)] + p10_w: list[float | None] + p50_w: list[float | None] + p90_w: list[float | None] + uncalibrated_p50_w: list[float | None] + calibrated_p50_w: list[float | None] | None + calibration_active: bool + """At least one source's calibration is applied.""" + stale: bool + """Every source's last good fetch is older than 12 h; the blend uses them anyway.""" + excluded_sources: list[str] + """Sources left out of the blend: older than 12 h while a fresher source has data.""" + weights: list[BlendWeightsView] + calibration: list[CalibrationView] + + +class PvForecastView(_View): + generated_at: AwareDatetime + hours: int + sources: list[PvSourceView] + blended: BlendedView | None + + +class AccuracySourceView(_View): + """Measured accuracy of one source; ``records`` only holds buckets with data. + + The source id ``blended`` is the blended forecast the planner used. + """ + + source_id: str + computed_at: AwareDatetime | None + window_days: int | None + records: list[AccuracyRecord] + + +class ForecastAccuracyView(_View): + """Accuracy per source and horizon bucket. + + ``bias_w = actual - forecast`` (positive = the source under-forecasts); ``n`` counts + 15-minute daylight intervals (forecast or measurement above 0 W; clipped ones excluded). + A record with ``n`` below ``min_samples`` is too thin to rely on. + """ + + min_samples: int + sources: list[AccuracySourceView] + + +__all__ = [ + "MAX_VIEW_HOURS", + "AccuracySourceView", + "BlendWeightsView", + "BlendedView", + "CalibrationView", + "ForecastAccuracyView", + "PvForecastView", + "PvSeriesView", + "PvSourceView", +] diff --git a/suncore/ha/__init__.py b/suncore/ha/__init__.py index bdc6b0e..92d67b6 100644 --- a/suncore/ha/__init__.py +++ b/suncore/ha/__init__.py @@ -52,6 +52,7 @@ parse_device_registry, parse_entity_registry, ) +from suncore.ha.solar_forecast import SolarForecast, fetch_solar_forecast from suncore.ha.transport import ( DirectTransport, SupervisorTransport, @@ -96,6 +97,7 @@ "ServiceCallerProvider", "ServiceResult", "SiteRole", + "SolarForecast", "StateContext", "SupervisorTransport", "Transport", @@ -106,6 +108,7 @@ "detect_site_role", "detect_template_role", "entity_candidates", + "fetch_solar_forecast", "is_candidate", "legacy_entities", "parse_area_registry", diff --git a/suncore/ha/connection.py b/suncore/ha/connection.py index 77dadb2..accaf08 100644 --- a/suncore/ha/connection.py +++ b/suncore/ha/connection.py @@ -237,6 +237,18 @@ async def energy_get_prefs(self) -> Mapping[str, Any] | None: """``energy/get_prefs``; ``None`` where HA answers ``not_found`` ("No prefs").""" ... + async def energy_solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + """``energy/solar_forecast`` (read only, not admin gated): ``{config_entry_id: {iso: Wh}}``. + + Only forecast integrations that are linked in the Energy dashboard's solar source appear + (``{}`` when none is, or the dashboard was never saved); an entry that returned ``None`` + is omitted by Home Assistant. The key is the **config entry id**, the value Wh per + period starting at the timestamp (period length differs per provider; infer it from the + key spacing). Entries whose values are not numbers are dropped here. Verified shape: + ``docs/research/ha-websocket-api.md`` "energy/solar_forecast". + """ + ... + async def subscribe_registry_updates(self, listener: RegistryListener) -> Unsubscribe: """Call ``listener(event_type)`` on every registry update event. diff --git a/suncore/ha/solar_forecast.py b/suncore/ha/solar_forecast.py new file mode 100644 index 0000000..11ce962 --- /dev/null +++ b/suncore/ha/solar_forecast.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +"""``energy/solar_forecast``: the forecasts of the integrations linked in the Energy dashboard. + +Read from ``home-assistant/core`` ``energy/websocket_api.py`` and ``energy/types.py`` +(``docs/research/ha-websocket-api.md`` "energy/solar_forecast"): + +* request ``{"type": "energy/solar_forecast"}``, no fields, not admin gated; +* result ``{"": {"wh_hours": {"": }}}``; an + integration that returned ``None`` is omitted, the result is ``{}`` when the dashboard has no + solar source or no forecast linked. + +**Selecting the entry.** The key is the *config entry id* (not the domain). The command does not +say which integration an entry belongs to, so a forecast source is bound to its entry by the +user's choice (``ForecastSourceConfig.config_entry_id``; the wizard offers the ids that +``energy/get_prefs`` lists in ``config_entry_solar_forecast``). Without a choice a source takes +the entry only if the result holds exactly one - SunCore never guesses between several. + +Only this module (and the connection) touches the command: ``tests/safety/test_boundary.py`` +rule 9 keeps ``energy_solar_forecast`` inside ``suncore.ha`` and ``suncore.sim``. +""" + +from __future__ import annotations + +import math +from collections.abc import Mapping +from typing import Any, Final + +import structlog + +from suncore.ha.connection import HAConnection + +log = structlog.get_logger(__name__) + +WH_HOURS: Final = "wh_hours" +"""Field of one entry (``SolarForecastType`` in ``energy/types.py``).""" + +SolarForecast = Mapping[str, Mapping[str, float]] +"""``{config_entry_id: {iso_timestamp: Wh}}`` (a value is a finite number).""" + + +def parse_solar_forecast(result: Mapping[str, Any]) -> dict[str, dict[str, float]]: + """The ``wh_hours`` of every entry; malformed entries and non-numeric values are dropped.""" + out: dict[str, dict[str, float]] = {} + for entry_id, body in result.items(): + hours = body.get(WH_HOURS) if isinstance(body, Mapping) else None + if not isinstance(hours, Mapping): + log.warning("ha.solar_forecast_entry_malformed", entry=str(entry_id)[:64]) + continue + out[str(entry_id)] = { + str(stamp): float(value) + for stamp, value in hours.items() + if isinstance(value, int | float) + and not isinstance(value, bool) + and math.isfinite(float(value)) + } + return out + + +async def fetch_solar_forecast(connection: HAConnection) -> SolarForecast: + """``energy/solar_forecast`` of the connection (the one place the adapters call it).""" + return await connection.energy_solar_forecast() + + +__all__ = ["WH_HOURS", "SolarForecast", "fetch_solar_forecast", "parse_solar_forecast"] diff --git a/suncore/ha/websocket.py b/suncore/ha/websocket.py index 6a2d47a..ed86ac4 100644 --- a/suncore/ha/websocket.py +++ b/suncore/ha/websocket.py @@ -99,6 +99,7 @@ action_rate_key, ) from suncore.ha.redact import DEFAULT_REGISTRY, SecretRegistry +from suncore.ha.solar_forecast import parse_solar_forecast from suncore.ha.transport import Transport log = structlog.get_logger(__name__) @@ -449,6 +450,13 @@ async def energy_get_prefs(self) -> Mapping[str, Any] | None: raise ProtocolError(msg) return cast("Mapping[str, Any]", result) + async def energy_solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + result = await self._command({"type": "energy/solar_forecast"}) + if not isinstance(result, Mapping): + msg = "energy/solar_forecast returned a non-object" + raise ProtocolError(msg) + return parse_solar_forecast(result) + async def subscribe_registry_updates(self, listener: RegistryListener) -> Unsubscribe: self._registry_listeners.append(listener) if self._state is ConnectionState.CONNECTED: diff --git a/suncore/sim/demo_config.py b/suncore/sim/demo_config.py index 6d5b3f2..2a5d130 100644 --- a/suncore/sim/demo_config.py +++ b/suncore/sim/demo_config.py @@ -36,6 +36,8 @@ AC_ID, FORECAST_TODAY_ID, FORECAST_TOMORROW_ID, + OPEN_METEO_TODAY_ID, + OPEN_METEO_TOMORROW_ID, POOL_POWER_ID, POOL_SWITCH_ID, PRICE_ID, @@ -181,7 +183,18 @@ async def seed_demo_config(config: ConfigStore, catalog: TemplateCatalog) -> Non "forecast_today": EntityRef(entity_id=FORECAST_TODAY_ID), "forecast_tomorrow": EntityRef(entity_id=FORECAST_TOMORROW_ID), }, - ) + ), + ForecastSourceConfig( + template_id="open_meteo_solar_forecast.pv", + entities={ + "energy_today": EntityRef(entity_id=OPEN_METEO_TODAY_ID), + "energy_tomorrow": EntityRef(entity_id=OPEN_METEO_TOMORROW_ID), + }, + ), + # Two entries are linked in the demo's Energy dashboard: the choice is explicit. + ForecastSourceConfig( + template_id="forecast_solar.pv", config_entry_id="sim_entry_forecast_solar" + ), ], ), ) diff --git a/suncore/sim/demo_devices.py b/suncore/sim/demo_devices.py index a86bd2a..9ec914f 100644 --- a/suncore/sim/demo_devices.py +++ b/suncore/sim/demo_devices.py @@ -22,7 +22,11 @@ * a Tibber-like price sensor ``sensor.sim_electricity_price`` (``EUR/kWh``, 15 min slots); * Solcast-like day sensors ``sensor.sim_solcast_forecast_today`` / ``..._tomorrow`` (kWh) whose ``detailedForecast`` attribute holds 30 min slots in kW average power (P50/P10/P90) - derived from the virtual house's PV model; + derived from the virtual house's PV model with deterministic noise + (``suncore.sim.forecasts``), Open-Meteo-like sensors + ``sensor.sim_open_meteo_energy_production_today`` / ``..._tomorrow`` (``wh_period_15m``) and a + template sensor ``sensor.sim_pv_forecast_template`` (points ``{start, power_w}``), all + refreshed hourly of house time; * kWh energy counters for grid, PV and battery so the fake Energy dashboard preferences have real statistics. @@ -53,7 +57,11 @@ WaterHeaterParams, build_device_set, ) -from suncore.sim.house import HouseState, clear_sky_w +from suncore.sim.forecasts import ( + SOLCAST_SLOT_MIN, + SimForecastModel, +) +from suncore.sim.house import HouseState from suncore.sim.scenario import Scenario AC_ID: Final = "climate.sim_living_room_ac" @@ -75,6 +83,9 @@ PRICE_ID: Final = "sensor.sim_electricity_price" FORECAST_TODAY_ID: Final = "sensor.sim_solcast_forecast_today" FORECAST_TOMORROW_ID: Final = "sensor.sim_solcast_forecast_tomorrow" +OPEN_METEO_TODAY_ID: Final = "sensor.sim_open_meteo_energy_production_today" +OPEN_METEO_TOMORROW_ID: Final = "sensor.sim_open_meteo_energy_production_tomorrow" +TEMPLATE_FORECAST_ID: Final = "sensor.sim_pv_forecast_template" GRID_IMPORT_KWH_ID: Final = "sensor.sim_grid_energy_import" GRID_EXPORT_KWH_ID: Final = "sensor.sim_grid_energy_export" PV_KWH_ID: Final = "sensor.sim_pv_energy" @@ -97,6 +108,9 @@ PRICE_ID, FORECAST_TODAY_ID, FORECAST_TOMORROW_ID, + OPEN_METEO_TODAY_ID, + OPEN_METEO_TOMORROW_ID, + TEMPLATE_FORECAST_ID, GRID_IMPORT_KWH_ID, GRID_EXPORT_KWH_ID, PV_KWH_ID, @@ -113,9 +127,6 @@ """Electrical power of the demo air conditioner in standby.""" POOL_PUMP_POWER_W: Final = 750.0 PRICE_CURRENCY: Final = "EUR" -SLOT_MIN: Final = 30 -TOMORROW_CLOUD_FACTOR: Final = 0.8 -"""Assumed cloud factor of the demo forecast for the next day.""" def _gauss(x: float, centre: float, sigma: float) -> float: @@ -238,13 +249,15 @@ class DemoDevices: _c: _Counters = field(default_factory=_Counters) _state: HouseState | None = None _forecast_key: tuple[Any, int] | None = None - _forecasts: tuple[DemoEntity, DemoEntity] | None = None + _forecasts: list[DemoEntity] | None = None + _model: SimForecastModel = field(init=False) _cloud: list[float] = field(default_factory=list) _user_ac_on: bool = False _user_pool_on: bool = False def __post_init__(self) -> None: self._cloud = self.scenario.cloud_series() + self._model = SimForecastModel(self.scenario, self._cloud_at) specs = [(spec.id, spec.physics) for spec in self.scenario.devices] self.is_default = not specs self.devices = build_device_set(specs) if specs else default_demo_devices() @@ -325,7 +338,7 @@ def entities(self) -> list[DemoEntity]: if state is None: return [] c = self._c - today, tomorrow = self._forecast_entities(state.time) + forecasts = self._forecast_entities(state.time) entities = [ DemoEntity(s.entity_id, s.state, s.attributes) for s in self.devices.states(state.time) ] @@ -340,8 +353,7 @@ def entities(self) -> list[DemoEntity]: "state_class": "measurement", }, ), - today, - tomorrow, + *forecasts, self._kwh(GRID_IMPORT_KWH_ID, "Sim grid energy import", c.grid_import_kwh), self._kwh(GRID_EXPORT_KWH_ID, "Sim grid energy export", c.grid_export_kwh), self._kwh(PV_KWH_ID, "Sim PV energy", c.pv_kwh), @@ -374,42 +386,32 @@ def _cloud_at(self, when: datetime) -> float: return 1.0 return self._cloud[min(max(index, 0), len(self._cloud) - 1)] - def _day_slots(self, midnight: datetime, *, tomorrow: bool) -> list[dict[str, Any]]: - pv = self.scenario.house.pv - slots: list[dict[str, Any]] = [] - for i in range(24 * 60 // SLOT_MIN): - start = midnight + timedelta(minutes=i * SLOT_MIN) - mid = start + timedelta(minutes=SLOT_MIN / 2) - clear_kw = clear_sky_w(mid, pv) / 1000 - cloud = TOMORROW_CLOUD_FACTOR if tomorrow else self._cloud_at(mid) - p50 = clear_kw * cloud - slots.append( - { - "period_start": start.isoformat(), - "pv_estimate": round(p50, 4), - "pv_estimate10": round(p50 * 0.6, 4), - "pv_estimate90": round(min(clear_kw, p50 * 1.15), 4), - } - ) - return slots + def solar_forecast(self, now: datetime) -> dict[str, dict[str, float]]: + """``energy/solar_forecast`` of the demo: the two entries linked in the Energy dashboard. - def _forecast_entities(self, now: datetime) -> tuple[DemoEntity, DemoEntity]: + ``sim_entry_forecast_solar``: hourly Wh, a zero at midnight dropped (Forecast.Solar); + ``sim_entry_solcast_solar``: half-hourly Wh of the P50 (Solcast's energy platform). + """ + return { + "sim_entry_forecast_solar": self._model.forecast_solar_wh_hours(now), + "sim_entry_solcast_solar": self._model.solcast_wh_hours( + now.replace(minute=0, second=0, microsecond=0), 48, now + ), + } + + def _forecast_entities(self, now: datetime) -> list[DemoEntity]: key = (now.date(), now.hour) if self._forecasts is not None and self._forecast_key == key: return self._forecasts midnight = now.replace(hour=0, minute=0, second=0, microsecond=0) + model = self._model result: list[DemoEntity] = [] - for entity_id, name, day, tomorrow in ( - (FORECAST_TODAY_ID, "Sim Solcast forecast today", midnight, False), - ( - FORECAST_TOMORROW_ID, - "Sim Solcast forecast tomorrow", - midnight + timedelta(days=1), - True, - ), + for entity_id, name, day in ( + (FORECAST_TODAY_ID, "Sim Solcast forecast today", midnight), + (FORECAST_TOMORROW_ID, "Sim Solcast forecast tomorrow", midnight + timedelta(days=1)), ): - slots = self._day_slots(day, tomorrow=tomorrow) - total_kwh = sum(s["pv_estimate"] for s in slots) * SLOT_MIN / 60 + slots = model.solcast_day(day, now) + total_kwh = sum(s["pv_estimate"] for s in slots) * SOLCAST_SLOT_MIN / 60 result.append( DemoEntity( entity_id, @@ -425,6 +427,42 @@ def _forecast_entities(self, now: datetime) -> tuple[DemoEntity, DemoEntity]: }, ) ) + for entity_id, name, day in ( + (OPEN_METEO_TODAY_ID, "Sim Open-Meteo energy production today", midnight), + ( + OPEN_METEO_TOMORROW_ID, + "Sim Open-Meteo energy production tomorrow", + midnight + timedelta(days=1), + ), + ): + wh_15m = model.open_meteo_day(day, now) + result.append( + DemoEntity( + entity_id, + f"{sum(wh_15m.values()) / 1000:.3f}", + { + "friendly_name": name, + "unit_of_measurement": "kWh", + "device_class": "energy", + "state_class": "total", + "wh_period_15m": wh_15m, + }, + ) + ) + points = model.template_points(midnight, 2, now) + template_kwh = sum(p["power_w"] for p in points) * 0.5 / 1000 + result.append( + DemoEntity( + TEMPLATE_FORECAST_ID, + f"{template_kwh:.2f}", + { + "friendly_name": "Sim PV forecast (template sensor)", + "unit_of_measurement": "kWh", + "device_class": "energy", + "forecast": points, + }, + ) + ) self._forecast_key = key - self._forecasts = (result[0], result[1]) - return self._forecasts + self._forecasts = result + return result diff --git a/suncore/sim/fake_ha.py b/suncore/sim/fake_ha.py index 41edd80..df575d2 100644 --- a/suncore/sim/fake_ha.py +++ b/suncore/sim/fake_ha.py @@ -80,6 +80,7 @@ from suncore.sim import registry from suncore.sim.demo_devices import ( FORECAST_TODAY_ID, + FORECAST_TOMORROW_ID, SIM_DEVICE_ENTITY_IDS, DemoDevices, sim_price, @@ -154,7 +155,9 @@ def demo_site_config() -> SiteConfig: """Site configuration mapping the simulator entities (signed grid, one PV, battery).""" return SiteConfig( grid=GridConfig(power=EntityRef(entity_id=GRID_ID)), - pv=[PvArray(name="PV", power=EntityRef(entity_id=PV_ID))], + # The simulator's plant: 8 kWp following sin(elevation), i.e. a horizontal plane + # (``PVParams`` defaults; the clear-sky model of the forecast calibration reads this). + pv=[PvArray(name="PV", power=EntityRef(entity_id=PV_ID), kwp=8.0, tilt_deg=0.0)], battery=BatteryConfig( power=EntityRef(entity_id=BATTERY_ID), soc=EntityRef(entity_id=SOC_ID) ), @@ -176,6 +179,25 @@ def _format(value: float) -> str: return f"{value + 0.0:.1f}" if abs(value) >= 0.05 else "0.0" +def _solcast_query_window(slots: list[Any], data: Mapping[str, Any] | None) -> list[Any]: + """The slots of ``query_forecast_data`` between ``start_date_time`` and ``end_date_time``. + + Both default to *now* in the real action (an empty range); the fake keeps every slot when a + bound is absent so older tests that call it without a range still get the day. + """ + if not data: + return slots + out = slots + start, end = data.get("start_date_time"), data.get("end_date_time") + if isinstance(start, str): + lo = datetime.fromisoformat(start) + out = [s for s in out if datetime.fromisoformat(s["period_start"]) >= lo] + if isinstance(end, str): + hi = datetime.fromisoformat(end) + out = [s for s in out if datetime.fromisoformat(s["period_start"]) < hi] + return out + + class FakeHAConnection: """Simulator-backed :class:`HAConnection`. @@ -408,6 +430,20 @@ def fire_registry_event(self, event_type: str = REGISTRY_EVENT_TYPES[0]) -> None except Exception: _log.exception("registry listener failed") + async def energy_solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + """``energy/solar_forecast``: the entries linked in the demo's Energy dashboard. + + ``sim_entry_forecast_solar`` (hourly Wh, a zero at midnight dropped, like Forecast.Solar) + and ``sim_entry_solcast_solar`` (half-hourly Wh of the P50). ``{}`` while the dashboard is + not configured (:meth:`energy_get_prefs` returns ``None``). + """ + self._require_connected() + if self.registry_error is not None: + raise self.registry_error + if not self._energy_prefs_configured or self._demo is None or self._house_state is None: + return {} + return self._demo.solar_forecast(self._house_state.time) + async def statistics_during_period( self, statistic_ids: Sequence[str], @@ -430,7 +466,7 @@ async def call_action_response( self, domain: str, service: str, - data: Mapping[str, Any] | None = None, # noqa: ARG002 + data: Mapping[str, Any] | None = None, *, target: Mapping[str, Any] | None = None, # noqa: ARG002 min_interval_s: float | None = None, @@ -452,9 +488,12 @@ async def call_action_response( prices = [{"start_time": t.isoformat(), "price": sim_price(t)} for t in slots] return {"prices": {"Home": prices}} if (domain, service) == ("solcast_solar", "query_forecast_data"): - today = self._published.get(FORECAST_TODAY_ID) - slots_data = today.attributes.get("detailedForecast", []) if today else [] - return {"data": list(slots_data)} + slots_data: list[Any] = [] + for entity_id in (FORECAST_TODAY_ID, FORECAST_TOMORROW_ID): + day = self._published.get(entity_id) + if day is not None: + slots_data.extend(day.attributes.get("detailedForecast", [])) + return {"data": _solcast_query_window(slots_data, data)} raise CommandError("not_found", f"Service {domain}.{service} not found.") def entity_metas(self) -> list[EntityMeta]: diff --git a/suncore/sim/forecasts.py b/suncore/sim/forecasts.py new file mode 100644 index 0000000..cd88747 --- /dev/null +++ b/suncore/sim/forecasts.py @@ -0,0 +1,153 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Forecast shapes of the demo: what Solcast, Open-Meteo Solar Forecast and Forecast.Solar +would publish for the virtual house (Phase 4 WP5). + +All sources derive from the simulator's PV model (clear-sky power times the cloud factor) and add +**deterministic** noise: the same ``seed`` and the same instants always give the same numbers, so +tests and the demo are reproducible. Each source has its own error character so the accuracy +statistics have something to show: + +* ``solcast``: unbiased, 6 % noise, P10 = 0.6 x P50 and P90 = min(clear sky, 1.15 x P50); +* ``open_meteo``: 6 % too low, 8 % noise, no quantiles; +* ``forecast_solar``: 4 % too high, 7 % noise, hourly Wh, only a zero hour at local midnight is + dropped (the real integration's ``energy`` platform, ``forecast_solar/energy.py``, keeps + ``val != 0 or (h, m, s) != (0, 0, 0)``); +* ``template``: the generic template's own contract, points ``{start, power_w}`` (30 min). + +The noise grows with the lead time (half at lead 0, full at 48 h), and a forecast issued in a +later hour differs slightly from an earlier one, like a real refresh. + +Shapes (all from ``docs/research/integrations.md``): Solcast ``detailedForecast`` entries +``{period_start, pv_estimate, pv_estimate10, pv_estimate90}`` in kW (average power of 30 min); +Open-Meteo ``wh_period_15m`` ``{interval start ISO: Wh per 15 min}``; Forecast.Solar +``wh_hours`` ``{ISO timestamp: Wh}`` of hourly periods. +""" + +from __future__ import annotations + +import random +from collections.abc import Callable +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import Any, Final + +from suncore.sim.house import clear_sky_w +from suncore.sim.scenario import Scenario + +FORECAST_SEED: Final = 20_261_016 +"""Seed of the demo forecast noise.""" + +TOMORROW_CLOUD_FACTOR: Final = 0.8 +"""Assumed cloud factor of the demo forecast for the following days.""" + +SOLCAST_SLOT_MIN: Final = 30 +OPEN_METEO_SLOT_MIN: Final = 15 +TEMPLATE_SLOT_MIN: Final = 30 + +_BIAS: Final = {"solcast": 1.0, "open_meteo": 0.94, "forecast_solar": 1.04, "template": 1.0} +_SIGMA: Final = {"solcast": 0.06, "open_meteo": 0.08, "forecast_solar": 0.07, "template": 0.05} + + +@dataclass(frozen=True, slots=True) +class SimForecastModel: + """The forecasts of the demo, as the integrations would publish them at ``now``.""" + + scenario: Scenario + cloud_at: Callable[[datetime], float] + seed: int = FORECAST_SEED + + def _factor(self, source: str, slot: datetime, now: datetime) -> float: + """Bias and noise of one slot as issued at ``now`` (deterministic).""" + lead_h = min(max((slot - now).total_seconds() / 3600, 0.0), 48.0) + sigma = _SIGMA[source] * (0.5 + 0.5 * lead_h / 48.0) + issue = now.replace(minute=0, second=0, microsecond=0).isoformat() + key = f"{self.seed}|{source}|{slot.isoformat()}|{issue}" + noise = random.Random(key).gauss(0.0, sigma) + return _BIAS[source] * min(max(1.0 + noise, 0.6), 1.4) + + def _powers_w( + self, source: str, start: datetime, minutes: int, now: datetime + ) -> tuple[float, float]: + """``(clear_sky_w, p50_w)`` of the slot ``[start, start + minutes)`` as seen at ``now``.""" + mid = start + timedelta(minutes=minutes / 2) + clear = clear_sky_w(mid, self.scenario.house.pv) + beyond_today = start.date() > now.date() + cloud = TOMORROW_CLOUD_FACTOR if beyond_today else self.cloud_at(mid) + p50 = min(clear, clear * cloud * self._factor(source, start, now)) + return clear, p50 + + # --- Solcast ------------------------------------------------------------------- + + def solcast_day(self, midnight: datetime, now: datetime) -> list[dict[str, Any]]: + """``detailedForecast`` of the local day starting at ``midnight`` (kW, 30 min).""" + slots: list[dict[str, Any]] = [] + for i in range(24 * 60 // SOLCAST_SLOT_MIN): + start = midnight + timedelta(minutes=i * SOLCAST_SLOT_MIN) + clear, p50 = self._powers_w("solcast", start, SOLCAST_SLOT_MIN, now) + p50_kw = p50 / 1000 + slots.append( + { + "period_start": start.isoformat(), + "pv_estimate": round(p50_kw, 4), + "pv_estimate10": round(p50_kw * 0.6, 4), + "pv_estimate90": round(min(clear / 1000, p50_kw * 1.15), 4), + } + ) + return slots + + def solcast_wh_hours(self, start: datetime, hours: int, now: datetime) -> dict[str, float]: + """The Solcast ``energy`` platform: half-hour keys, ``kW * 500`` Wh (P50, integers).""" + out: dict[str, float] = {} + for i in range(hours * 60 // SOLCAST_SLOT_MIN): + slot = start + timedelta(minutes=i * SOLCAST_SLOT_MIN) + _, p50 = self._powers_w("solcast", slot, SOLCAST_SLOT_MIN, now) + out[slot.isoformat()] = float(round(p50 / 1000 * 500)) + return out + + # --- Open-Meteo ---------------------------------------------------------------- + + def open_meteo_day(self, midnight: datetime, now: datetime) -> dict[str, float]: + """``wh_period_15m`` of the local day: Wh per 15 minutes, keyed by the interval start.""" + out: dict[str, float] = {} + for i in range(24 * 60 // OPEN_METEO_SLOT_MIN): + start = midnight + timedelta(minutes=i * OPEN_METEO_SLOT_MIN) + _, p50 = self._powers_w("open_meteo", start, OPEN_METEO_SLOT_MIN, now) + out[start.isoformat()] = round(p50 * OPEN_METEO_SLOT_MIN / 60, 3) + return out + + # --- Forecast.Solar -------------------------------------------------------------- + + def forecast_solar_wh_hours(self, now: datetime, hours: int = 48) -> dict[str, float]: + """Hourly Wh from the current hour on; only a zero hour starting at local midnight is + dropped (like ``forecast_solar/energy.py``).""" + first = now.replace(minute=0, second=0, microsecond=0) + out: dict[str, float] = {} + for i in range(hours): + start = first + timedelta(hours=i) + _, p50 = self._powers_w("forecast_solar", start, 60, now) + wh = float(round(p50)) + midnight = (start.hour, start.minute, start.second) == (0, 0, 0) + if not (wh == 0 and midnight): + out[start.isoformat()] = wh + return out + + # --- generic template sensor ------------------------------------------------------ + + def template_points(self, midnight: datetime, days: int, now: datetime) -> list[dict[str, Any]]: + """Points ``{start, power_w}`` (30 min) of ``days`` local days.""" + points: list[dict[str, Any]] = [] + for i in range(days * 24 * 60 // TEMPLATE_SLOT_MIN): + start = midnight + timedelta(minutes=i * TEMPLATE_SLOT_MIN) + _, p50 = self._powers_w("template", start, TEMPLATE_SLOT_MIN, now) + points.append({"start": start.isoformat(), "power_w": round(p50, 1)}) + return points + + +__all__ = [ + "FORECAST_SEED", + "OPEN_METEO_SLOT_MIN", + "SOLCAST_SLOT_MIN", + "TEMPLATE_SLOT_MIN", + "TOMORROW_CLOUD_FACTOR", + "SimForecastModel", +] diff --git a/suncore/sim/registry.py b/suncore/sim/registry.py index 24a300c..506d25e 100644 --- a/suncore/sim/registry.py +++ b/suncore/sim/registry.py @@ -41,6 +41,7 @@ DEV_WATER_HEATER_RELAYS: Final = "sim_dev_water_heater_relays" DEV_TIBBER: Final = "sim_dev_tibber" DEV_SOLCAST: Final = "sim_dev_solcast" +DEV_OPEN_METEO: Final = "sim_dev_open_meteo" @dataclass(frozen=True, slots=True) @@ -124,6 +125,25 @@ def _sim(entity_id: str, name: str) -> RegEntity: None, "total_kwh_forecast_tomorrow", ), + # Open-Meteo Solar Forecast: translation key assumed to equal the sensor key (UNVERIFIED). + RegEntity( + d.OPEN_METEO_TODAY_ID, + "open_meteo_solar_forecast", + "Energy production - today", + DEV_OPEN_METEO, + None, + "energy_production_today", + ), + RegEntity( + d.OPEN_METEO_TOMORROW_ID, + "open_meteo_solar_forecast", + "Energy production - tomorrow", + DEV_OPEN_METEO, + None, + "energy_production_tomorrow", + ), + # A user's Home Assistant template sensor that exposes the generic forecast attribute. + RegEntity(d.TEMPLATE_FORECAST_ID, "template", "PV forecast", DEV_HOUSE), ) AREAS: Final[tuple[tuple[str, str], ...]] = ((AREA_LIVING, "Living room"), (AREA_GARDEN, "Garden")) @@ -153,12 +173,21 @@ def _sim(entity_id: str, name: str) -> RegEntity: ), (DEV_TIBBER, "Home", "Tibber", "Price Sensor", None, "service"), (DEV_SOLCAST, "Solcast PV Forecast", "BJReplay", "Solcast PV Forecast", None, "service"), + ( + DEV_OPEN_METEO, + "Solar production forecast", + "Open-Meteo", + "Solar production forecast", + None, + "service", + ), ) _DEVICE_DOMAIN: Final[dict[str, str]] = { DEV_AC: "lg_thinq", DEV_AC_B: "smartthinq_sensors", DEV_TIBBER: "tibber", DEV_SOLCAST: "solcast_solar", + DEV_OPEN_METEO: "open_meteo_solar_forecast", } @@ -303,7 +332,10 @@ def energy_prefs(battery_capacity_kwh: float) -> dict[str, Any]: "type": "solar", "stat_energy_from": d.PV_KWH_ID, "stat_rate": "sensor.sim_pv_power", - "config_entry_solar_forecast": ["sim_entry_solcast_solar"], + "config_entry_solar_forecast": [ + "sim_entry_solcast_solar", + "sim_entry_forecast_solar", + ], }, { "type": "battery", diff --git a/suncore/storage/learned.py b/suncore/storage/learned.py index b693c27..97b4e85 100644 --- a/suncore/storage/learned.py +++ b/suncore/storage/learned.py @@ -27,9 +27,17 @@ NS_LOAD_PROFILE: Final = "load_profile" NS_BLEND_WEIGHTS: Final = "blend_weights" NS_ML_STATE: Final = "ml_state" +NS_FORECAST_ACCURACY: Final = "forecast_accuracy" KNOWN_NAMESPACES: Final[frozenset[str]] = frozenset( - {NS_FORECAST_BIAS, NS_NET_ERROR, NS_LOAD_PROFILE, NS_BLEND_WEIGHTS, NS_ML_STATE} + { + NS_FORECAST_BIAS, + NS_NET_ERROR, + NS_LOAD_PROFILE, + NS_BLEND_WEIGHTS, + NS_ML_STATE, + NS_FORECAST_ACCURACY, + } ) """The namespaces Phase 4 uses; other well-formed names are accepted (later phases).""" @@ -138,6 +146,7 @@ def _decode(row: tuple[object, ...]) -> LearnedParam | None: __all__ = [ "KNOWN_NAMESPACES", "NS_BLEND_WEIGHTS", + "NS_FORECAST_ACCURACY", "NS_FORECAST_BIAS", "NS_LOAD_PROFILE", "NS_ML_STATE", diff --git a/suncore/templates/__init__.py b/suncore/templates/__init__.py index 49cae6a..455fa61 100644 --- a/suncore/templates/__init__.py +++ b/suncore/templates/__init__.py @@ -24,6 +24,7 @@ AttributeSeries, BatteryTemplate, DeviceTemplate, + EnergyDashboardSeries, EntityMatcher, ForecastTemplate, RegionTemplate, @@ -40,6 +41,7 @@ "BatteryTemplate", "BuiltinTemplateError", "DeviceTemplate", + "EnergyDashboardSeries", "EntityMatcher", "ForecastTemplate", "RegionTemplate", diff --git a/suncore/templates/models.py b/suncore/templates/models.py index 63c92c6..bf12d4b 100644 --- a/suncore/templates/models.py +++ b/suncore/templates/models.py @@ -304,8 +304,42 @@ def _allowed(self) -> Self: SeriesSource = Annotated[AttributeSeries | ActionSeries, Field(discriminator="source")] +class EnergyDashboardSeries(_Frozen): + """Series read from the WebSocket command ``energy/solar_forecast`` (Phase 4, forecasts only). + + The command returns ``{config_entry_id: {"wh_hours": {iso_timestamp: Wh}}}`` for the forecast + integrations linked in the Energy dashboard (Forecast.Solar in Home Assistant core). The + template names no entity: the user binds the source to a config entry + (``ForecastSourceConfig.config_entry_id``), or SunCore takes the only entry there is. + ``fields`` gives only ``value_unit`` (energy per period; no quantiles). The length of a + period is inferred from the spacing of the timestamps; the template's ``resolution_min`` + is the expected length (used for a series of one point). + """ + + source: Literal["energy_solar_forecast"] + fields: SeriesFields + missing_is_zero: bool = False + """The integration drops periods of value 0 (Forecast.Solar: only a 0 Wh period that starts + exactly at 00:00:00, ``forecast_solar/energy.py`` keeps ``val != 0 or not midnight``), so a + period missing between two known ones is 0 Wh, not unknown.""" + + @model_validator(mode="after") + def _fields(self) -> Self: + f = self.fields + if any(x is not None for x in (f.timestamp_field, f.value_field, f.p10_field, f.p90_field)): + msg = "series.fields: `energy_solar_forecast` takes only `value_unit`" + raise ValueError(msg) + return self + + +ForecastSeriesSource = Annotated[ + AttributeSeries | ActionSeries | EnergyDashboardSeries, Field(discriminator="source") +] + + def _check_series_roles( - series: AttributeSeries | ActionSeries, roles: dict[RoleName, RoleSpec] + series: AttributeSeries | ActionSeries | EnergyDashboardSeries, + roles: dict[RoleName, RoleSpec], ) -> None: if isinstance(series, AttributeSeries): for role in series.roles: @@ -506,7 +540,10 @@ class ForecastTemplate(TemplateHeader): kind: Literal["forecast"] forecast_kind: Literal["pv", "load"] roles: dict[RoleName, RoleSpec] = Field(default_factory=dict) - series: SeriesSource + series: ForecastSeriesSource + fallback_series: ActionSeries | None = None + """Used when the primary ``series`` yields nothing (Solcast: the attribute is switched off, + then the read-only action ``query_forecast_data``).""" resolution_min: Annotated[int, Field(ge=1, le=1440)] @model_validator(mode="after") @@ -516,13 +553,16 @@ def _forecast(self) -> Self: raise ValueError(msg) _check_roles(self.roles, required=isinstance(self.series, AttributeSeries)) _check_series_roles(self.series, self.roles) - unit = parse_unit(self.series.fields.value_unit) - if unit not in _FORECAST_UNITS: - msg = ( - f"series.fields.value_unit: `{self.series.fields.value_unit}` must be one of " - "W, kW, Wh, kWh (power = average over the slot, energy = per slot)" - ) - raise ValueError(msg) + for name, series in (("series", self.series), ("fallback_series", self.fallback_series)): + if series is None: + continue + unit = parse_unit(series.fields.value_unit) + if unit not in _FORECAST_UNITS: + msg = ( + f"{name}.fields.value_unit: `{series.fields.value_unit}` must be one of " + "W, kW, Wh, kWh (power = average over the slot, energy = per slot)" + ) + raise ValueError(msg) fields = self.series.fields has_quantiles = fields.p10_field is not None and fields.p90_field is not None _need( @@ -825,9 +865,11 @@ def message_keys(self) -> list[str]: "BatteryTemplate", "DeviceDefaults", "DeviceTemplate", + "EnergyDashboardSeries", "EntityMatcher", "FeedInTier", "ForceChargeSwitchControl", + "ForecastSeriesSource", "ForecastTemplate", "ModeSelectControl", "NegativePriceRule", diff --git a/templates/README.md b/templates/README.md index b1cfc4a..04f25ce 100644 --- a/templates/README.md +++ b/templates/README.md @@ -26,7 +26,10 @@ drift). | device | `generic.water_heater_steps` | water heater with power levels through one select entity, optional tank probe | generic, simulator only (`UNVERIFIED`: no hardware) | | device | `generic.water_heater_relays` | water heater with 1-3 heating elements switched by relays | generic, simulator only (`UNVERIFIED`: no hardware) | | tariff | `tibber.price` | Tibber dynamic price, curve from `tibber.get_prices` | from integration source | -| forecast | `solcast.pv` | Solcast PV Forecast (HACS), `detailedForecast` P10/P50/P90 | from integration source | +| forecast | `solcast.pv` | Solcast PV Forecast (HACS), `detailedForecast` P10/P50/P90 of today, tomorrow, d3, d4; fallback action `query_forecast_data` | from integration source (tag v4.6.1) | +| forecast | `forecast_solar.pv` | Forecast.Solar (core), hourly Wh through `energy/solar_forecast` (only if linked in the Energy dashboard), P50 only | from integration source, `UNVERIFIED` timestamp semantics | +| forecast | `open_meteo_solar_forecast.pv` | Open-Meteo Solar Forecast (HACS), `wh_period_15m` (Wh per 15 min), P50 only | from integration source (tag 0.1.33) | +| forecast | `generic.pv_forecast` | any sensor attribute with points `{start, power_w}` (copy and adapt for others) | SunCore's own contract | | region | `de` | Germany: EUR, `Europe/Berlin`, no remuneration at negative prices for new PV | law text `UNVERIFIED` here | | region | `generic` | neutral international profile | – | @@ -76,6 +79,19 @@ unverified: [] # facts you could not verify; required if tes Power units in a forecast mean the average power over the slot, energy units the energy per slot. When the attribute is a mapping, `list_path` (dotted) leads to the list of points; the period of a point is `resolution_min`. +- Forecast templates (Phase 4): `series.shape` is `list` (objects named by `fields`) or + `mapping` (`{iso_timestamp: value}`, `fields` holds only `value_unit`); a point is the + value of the period that **starts** at its timestamp. A forecast template may add + `fallback_series` (an allowlisted action) that is used when the attributes are missing. + `series.source: energy_solar_forecast` reads the Energy dashboard's `energy/solar_forecast` + (Forecast.Solar): the template names no entity, the user binds the source to a config entry + (`forecast.sources[].config_entry_id`; without one the only entry the command returns is + taken, several entries without a choice make the source report `ambiguous_entry`). + `missing_is_zero: true` reads a period missing between two known ones as 0 (Forecast.Solar's + integration drops a zero value that starts exactly at midnight). For an attribute or field + names the built-in templates do not cover, copy + `generic.pv_forecast` into `/data/templates` and change `attribute`, `fields`, `shape`, + `list_path` and `resolution_min`. - Battery templates (`kind: battery`) list the readable and writable roles (for example `soc`, `power`, `min_soc_number`, `max_charge_number`, `max_discharge_number`, `mode_select`, `force_charge_switch`) and in `controls` what each writable role @@ -128,3 +144,4 @@ Every addition is recorded below with the SunCore version that introduced it. | 0.3.0 (unreleased, Phase 2) | `region.currency` is optional for `country: generic` (`None` = currency of the Home Assistant configuration). | | 0.4.0 (unreleased, Phase 3) | New `defaults.control` kind `relays` (additive union member, ADR-0012 policy): 1–3 switch roles (`roles`) and `steps`, each step naming the relays that are on (`relays`) and the power it draws (`power_w`, strictly increasing, relay sets distinct). `DeviceConfig` accepts the same control. Used by `generic.water_heater_relays`. | | 0.5.0 (unreleased, Phase 4) | New template kind `battery` (`battery.schema.json`): `roles`, `controls` (union on `kind`: `min_soc`, `max_charge`, `max_discharge`, `grid_charge_power` numbers; `mode` select with option meanings; `force_charge` switch), optional `settle_s`. New capabilities `battery_soc`, `min_soc_limit`, `charge_limit`, `discharge_limit`, `grid_charge`, `battery_modes`. `series` of `source: attribute` takes an optional `list_path` for the generic attribute adapter and an optional `shape` (`list` default, or `mapping` for `{iso_timestamp: value}` attributes such as Forecast.Solar `wh_hours` and Open-Meteo Solar Forecast `wh_period_15m`; `fields` then holds only `value_unit`, so `fields.timestamp_field`/`value_field` became optional and are required only for lists of point objects). Tariff `price_unit` and `price_series.fields.value_unit` may be per MWh (`{currency}/MWh`, `EUR/MWh`); the value unit's currency must match `price_unit`'s (or be `{currency}`), and adapters convert to currency/kWh (`price_unit_factor`: MWh / 1000). Battery mode option `value` is kept verbatim (no whitespace stripping). HT/NT needs no template change: time windows and prices are contract data in the `tariff` configuration section (v3). | +| 0.5.0 (unreleased, Phase 4, PV forecasts) | Forecast templates: `series.source: energy_solar_forecast` (new union member, `fields` holds only `value_unit`, optional `missing_is_zero`) for the Energy dashboard's `energy/solar_forecast`; optional `fallback_series` (an `action` series) used when the attribute yields nothing. New templates `forecast_solar.pv`, `open_meteo_solar_forecast.pv`, `generic.pv_forecast`; `solcast.pv` gained the optional roles `forecast_d3`, `forecast_d4` and the fallback action `query_forecast_data`. | diff --git a/templates/forecast/forecast_solar.pv.yaml b/templates/forecast/forecast_solar.pv.yaml new file mode 100644 index 0000000..1f8f9d5 --- /dev/null +++ b/templates/forecast/forecast_solar.pv.yaml @@ -0,0 +1,40 @@ +# yaml-language-server: $schema=../schema/forecast.schema.json +# Forecast.Solar (Home Assistant core, domain `forecast_solar`). The integration has no series +# attribute; its hourly Wh per period is only available through the Energy dashboard's WebSocket +# command `energy/solar_forecast`, and only when the integration is linked as the forecast of a +# solar source there. No P10/P90. +id: forecast_solar.pv +kind: forecast +schema_version: 1 +name: template.forecast_solar.pv.name +source: + - title: "home-assistant/core forecast_solar/energy.py (async_get_solar_forecast, wh_hours, zero filter)" + url: "https://github.com/home-assistant/core/blob/5cfe9aad7cb428c2c2ebb449719ae81976780de1/homeassistant/components/forecast_solar/energy.py" + date: 2026-10-16 + - title: "home-assistant/core energy/websocket_api.py (energy/solar_forecast: which entries, result keyed by config entry id)" + url: "https://github.com/home-assistant/core/blob/5cfe9aad7cb428c2c2ebb449719ae81976780de1/homeassistant/components/energy/websocket_api.py" + date: 2026-10-16 + - title: "home-assistant/core energy/types.py (SolarForecastType = {wh_hours})" + url: "https://github.com/home-assistant/core/blob/5cfe9aad7cb428c2c2ebb449719ae81976780de1/homeassistant/components/energy/types.py" + date: 2026-10-16 + - title: "home-assistant/core forecast_solar/coordinator.py (update interval hourly, 30 min with an API key)" + url: "https://github.com/home-assistant/core/blob/5cfe9aad7cb428c2c2ebb449719ae81976780de1/homeassistant/components/forecast_solar/coordinator.py" + date: 2026-10-16 +tested_with: [] +capabilities: [pv_curve] +unverified: + - "Read from the integration source (core 5cfe9aad, forecast-solar 5.0.1) only; not observed on a running Home Assistant." + - "Whether the vendor API labels `watt_hours_period` keys as period start or end (api.forecast.solar unreachable); the HA library, and so SunCore, treats a key as the start." + - "Period length: hourly in the free tier (inferred from the key spacing); finer periods in paid plans are unverified." + - "The integration drops only a 0 Wh entry starting exactly at 00:00:00 (`energy.py`: `val != 0 or not midnight`); every other zero is listed." + - "A period missing between two known ones is read as 0 Wh (`missing_is_zero`); before the first and after the last entry nothing is assumed." + - "Whether the vendor API itself ever omits a non-zero period is unverified." + - "Whether the vendor API emits off-grid keys (sunrise/sunset times): such a key is dropped at 0 Wh, else added to its enclosing period." + - "Listed by `energy/solar_forecast` only if linked in the Energy dashboard; the user picks the config entry (`config_entry_id`) unless there is exactly one." +forecast_kind: pv +series: + source: energy_solar_forecast + missing_is_zero: true + fields: + value_unit: Wh +resolution_min: 60 diff --git a/templates/forecast/generic.pv_forecast.yaml b/templates/forecast/generic.pv_forecast.yaml new file mode 100644 index 0000000..a6644c7 --- /dev/null +++ b/templates/forecast/generic.pv_forecast.yaml @@ -0,0 +1,36 @@ +# yaml-language-server: $schema=../schema/forecast.schema.json +# Generic PV forecast from the attribute of any sensor (typically a Home Assistant template +# sensor that exposes a list of points). It describes a contract SunCore defines, not an +# integration: each point is an object `{start: , power_w: }` and +# `power_w` is the average power over the `resolution_min` minutes from `start`. For another +# attribute or other field names copy this file into /data/templates, change `attribute`, +# `fields`, `shape` (`mapping` for an object {ISO timestamp: value}), `list_path` and +# `resolution_min`, and give it a new id. +id: generic.pv_forecast +kind: forecast +schema_version: 1 +name: template.generic.pv_forecast.name +source: + - title: "SunCore templates README (series: attribute, shape list or mapping, list_path)" + url: "https://github.com/pxljake/suncore/blob/main/templates/README.md" + date: 2026-10-16 +tested_with: [] +capabilities: [pv_curve] +unverified: + - "Defines SunCore's own contract (attribute `forecast`, points `{start, power_w}`); it matches no existing integration and is meant as a starting point for a user template." +forecast_kind: pv +roles: + forecast: + matcher: + domain: sensor + platforms: [template] + description: template.generic.pv_forecast.role.forecast +series: + source: attribute + roles: [forecast] + attribute: forecast + fields: + timestamp_field: start + value_field: power_w + value_unit: W +resolution_min: 30 diff --git a/templates/forecast/open_meteo_solar_forecast.pv.yaml b/templates/forecast/open_meteo_solar_forecast.pv.yaml new file mode 100644 index 0000000..65b72e3 --- /dev/null +++ b/templates/forecast/open_meteo_solar_forecast.pv.yaml @@ -0,0 +1,71 @@ +# yaml-language-server: $schema=../schema/forecast.schema.json +# Open-Meteo Solar Forecast (HACS `rany2/ha-open-meteo-solar-forecast`, domain +# `open_meteo_solar_forecast`). The sensors `energy_production_today`, `_tomorrow`, `_d2` ... +# carry `wh_period_15m` = {interval start (ISO 8601 with offset): Wh per 15 minutes}. A single +# deterministic estimate (no P10/P90). +id: open_meteo_solar_forecast.pv +kind: forecast +schema_version: 1 +name: template.open_meteo_solar_forecast.pv.name +source: + - title: "ha-open-meteo-solar-forecast 0.1.33 sensor.py (energy_production_* sensors, wh_period_15m attribute)" + url: "https://github.com/rany2/ha-open-meteo-solar-forecast/blob/7af5d681c3ffeb7d8f678fd4eadd7f5aebd2a1ac/custom_components/open_meteo_solar_forecast/sensor.py" + date: 2026-10-16 + - title: "ha-open-meteo-solar-forecast 0.1.33 const.py (ATTR_WH_PERIOD_15M)" + url: "https://github.com/rany2/ha-open-meteo-solar-forecast/blob/7af5d681c3ffeb7d8f678fd4eadd7f5aebd2a1ac/custom_components/open_meteo_solar_forecast/const.py" + date: 2026-10-16 + - title: "ha-open-meteo-solar-forecast 0.1.33 coordinator.py (update interval 30 min)" + url: "https://github.com/rany2/ha-open-meteo-solar-forecast/blob/7af5d681c3ffeb7d8f678fd4eadd7f5aebd2a1ac/custom_components/open_meteo_solar_forecast/coordinator.py" + date: 2026-10-16 +tested_with: [] +capabilities: [pv_curve] +unverified: + - "Read from the integration source (tag 0.1.33) only; not observed on a running Home Assistant." + - "The sensors' translation key is assumed to equal the description key (`energy_production_today` ...); the research names only the key and the entity id pattern." + - "The state unit is Wh with kWh suggested for display; the matcher accepts both, the adapter reads the attribute, not the state." + - "Keys carry a fixed UTC offset of the plant location; the day boundary for a plant in another time zone than Home Assistant's is unverified." + - "Open-Meteo data is CC BY 4.0 (attribution in the UI); the separate terms page was unreachable." +forecast_kind: pv +roles: + energy_today: + matcher: + domain: sensor + platforms: [open_meteo_solar_forecast] + translation_key: energy_production_today + device_class: energy + units: [Wh, kWh] + description: template.open_meteo_solar_forecast.pv.role.energy_today + energy_tomorrow: + matcher: + domain: sensor + platforms: [open_meteo_solar_forecast] + translation_key: energy_production_tomorrow + device_class: energy + units: [Wh, kWh] + description: template.open_meteo_solar_forecast.pv.role.energy_tomorrow + energy_d2: + required: false + matcher: + domain: sensor + platforms: [open_meteo_solar_forecast] + translation_key: energy_production_d2 + device_class: energy + units: [Wh, kWh] + description: template.open_meteo_solar_forecast.pv.role.energy_d2 + energy_d3: + required: false + matcher: + domain: sensor + platforms: [open_meteo_solar_forecast] + translation_key: energy_production_d3 + device_class: energy + units: [Wh, kWh] + description: template.open_meteo_solar_forecast.pv.role.energy_d3 +series: + source: attribute + roles: [energy_today, energy_tomorrow, energy_d2, energy_d3] + attribute: wh_period_15m + shape: mapping + fields: + value_unit: Wh +resolution_min: 15 diff --git a/templates/forecast/solcast.pv.yaml b/templates/forecast/solcast.pv.yaml index ffd49c6..f5cb4fa 100644 --- a/templates/forecast/solcast.pv.yaml +++ b/templates/forecast/solcast.pv.yaml @@ -1,28 +1,33 @@ # yaml-language-server: $schema=../schema/forecast.schema.json # Solcast PV Forecast (HACS `BJReplay/ha-solcast-solar`, domain `solcast_solar`). -# The day sensors carry the half-hour curve in the attribute `detailedForecast` -# (average power in kW per slot, P50 / P10 / P90). +# The day sensors (today, tomorrow, d3 ...) carry the half-hour curve of their whole local day in +# the attribute `detailedForecast` (average power in kW per slot; P50 / P10 / P90). If the user +# switched that attribute off, the read-only action `solcast_solar.query_forecast_data` +# (does not consume Solcast API calls) delivers the same entries. id: solcast.pv kind: forecast schema_version: 1 name: template.solcast.pv.name source: - - title: "ha-solcast-solar forecast.py (detailedForecast slots, kW average power)" - url: "https://github.com/BJReplay/ha-solcast-solar/blob/3ae22af/custom_components/solcast_solar/forecast.py" - date: 2026-10-13 - - title: "ha-solcast-solar sensor.py (day sensors, units, device classes)" - url: "https://github.com/BJReplay/ha-solcast-solar/blob/3ae22af/custom_components/solcast_solar/sensor.py" - date: 2026-10-13 - - title: "ha-solcast-solar const.py (attribute and field names)" - url: "https://github.com/BJReplay/ha-solcast-solar/blob/3ae22af/custom_components/solcast_solar/const.py" - date: 2026-10-13 + - title: "ha-solcast-solar v4.6.1 forecast.py (detailedForecast slots, kW average power, P10/P90)" + url: "https://github.com/BJReplay/ha-solcast-solar/blob/34e1d007e9d4a9b7a783520d723fcd05adb2ff9a/custom_components/solcast_solar/forecast.py" + date: 2026-10-16 + - title: "ha-solcast-solar v4.6.1 const.py (attribute and field names, forecast_day_entities)" + url: "https://github.com/BJReplay/ha-solcast-solar/blob/34e1d007e9d4a9b7a783520d723fcd05adb2ff9a/custom_components/solcast_solar/const.py" + date: 2026-10-16 + - title: "ha-solcast-solar v4.6.1 coordinator.py (day sensors today, tomorrow, d3 ...)" + url: "https://github.com/BJReplay/ha-solcast-solar/blob/34e1d007e9d4a9b7a783520d723fcd05adb2ff9a/custom_components/solcast_solar/coordinator.py" + date: 2026-10-16 + - title: "ha-solcast-solar v4.6.1 actions.py and services.yaml (query_forecast_data, response data[])" + url: "https://github.com/BJReplay/ha-solcast-solar/blob/34e1d007e9d4a9b7a783520d723fcd05adb2ff9a/custom_components/solcast_solar/actions.py" + date: 2026-10-16 tested_with: [] capabilities: [pv_curve, pv_quantiles] unverified: - - "Read from the integration source (version 4.6.2, main 3ae22af) only; not observed on a running Home Assistant." - - "`detailedForecast` is only present when the option is on (default on); extra keys in the slots may appear with advanced options." - - "Day 3 to 7 sensors exist depending on an option and are not used by this template; the default number of day sensors is unknown." - - "Values are dampened when dampening is enabled in the integration." + - "Read from the integration source (tag v4.6.1) only; not observed on a running Home Assistant." + - "`detailedForecast` is only present when the option is on (default on); otherwise the action `query_forecast_data` is used (no Solcast API calls either way)." + - "Sensors d3 to d7 exist by default (option `forecast_day_entities`); only d3 and d4 are used (48 h planning). Their keys `total_kwh_forecast_d3` ... come from the research notes, not a running system." + - "Values are dampened when dampening is enabled in the integration (the action can return undampened values; SunCore does not ask for them)." forecast_kind: pv roles: forecast_today: @@ -41,9 +46,27 @@ roles: device_class: energy units: [kWh] description: template.solcast.pv.role.forecast_tomorrow + forecast_d3: + required: false + matcher: + domain: sensor + platforms: [solcast_solar] + translation_key: total_kwh_forecast_d3 + device_class: energy + units: [kWh] + description: template.solcast.pv.role.forecast_d3 + forecast_d4: + required: false + matcher: + domain: sensor + platforms: [solcast_solar] + translation_key: total_kwh_forecast_d4 + device_class: energy + units: [kWh] + description: template.solcast.pv.role.forecast_d4 series: source: attribute - roles: [forecast_today, forecast_tomorrow] + roles: [forecast_today, forecast_tomorrow, forecast_d3, forecast_d4] attribute: detailedForecast fields: timestamp_field: period_start @@ -51,4 +74,18 @@ series: p10_field: pv_estimate10 p90_field: pv_estimate90 value_unit: kW +fallback_series: + source: action + domain: solcast_solar + service: query_forecast_data + start_field: start_date_time + end_field: end_date_time + response_path: data + fields: + timestamp_field: period_start + value_field: pv_estimate + p10_field: pv_estimate10 + p90_field: pv_estimate90 + value_unit: kW + max_horizon_h: 96 resolution_min: 30 diff --git a/templates/schema/forecast.schema.json b/templates/schema/forecast.schema.json index 267aee7..e09b09c 100644 --- a/templates/schema/forecast.schema.json +++ b/templates/schema/forecast.schema.json @@ -164,6 +164,31 @@ "title": "Capability", "type": "string" }, + "EnergyDashboardSeries": { + "additionalProperties": false, + "description": "Series read from the WebSocket command ``energy/solar_forecast`` (Phase 4, forecasts only).\n\nThe command returns ``{config_entry_id: {\"wh_hours\": {iso_timestamp: Wh}}}`` for the forecast\nintegrations linked in the Energy dashboard (Forecast.Solar in Home Assistant core). The\ntemplate names no entity: the user binds the source to a config entry\n(``ForecastSourceConfig.config_entry_id``), or SunCore takes the only entry there is.\n``fields`` gives only ``value_unit`` (energy per period; no quantiles). The length of a\nperiod is inferred from the spacing of the timestamps; the template's ``resolution_min``\nis the expected length (used for a series of one point).", + "properties": { + "fields": { + "$ref": "#/$defs/SeriesFields" + }, + "missing_is_zero": { + "default": false, + "title": "Missing Is Zero", + "type": "boolean" + }, + "source": { + "const": "energy_solar_forecast", + "title": "Source", + "type": "string" + } + }, + "required": [ + "source", + "fields" + ], + "title": "EnergyDashboardSeries", + "type": "object" + }, "EntityMatcher": { "additionalProperties": false, "description": "Role matcher; every given criterion must match, ``domain`` is mandatory.\n\n``units`` lists acceptable ``unit_of_measurement`` strings (exact, case-sensitive);\n``unit_pattern`` is a regular expression matched with ``re.fullmatch`` on the stripped\nunit. ``translation_key`` / ``translation_key_pattern`` likewise. Exact and pattern\nforms exclude each other.", @@ -463,6 +488,17 @@ "title": "Capabilities", "type": "array" }, + "fallback_series": { + "anyOf": [ + { + "$ref": "#/$defs/ActionSeries" + }, + { + "type": "null" + } + ], + "default": null + }, "forecast_kind": { "enum": [ "pv", @@ -525,7 +561,8 @@ "discriminator": { "mapping": { "action": "#/$defs/ActionSeries", - "attribute": "#/$defs/AttributeSeries" + "attribute": "#/$defs/AttributeSeries", + "energy_solar_forecast": "#/$defs/EnergyDashboardSeries" }, "propertyName": "source" }, @@ -535,6 +572,9 @@ }, { "$ref": "#/$defs/ActionSeries" + }, + { + "$ref": "#/$defs/EnergyDashboardSeries" } ], "title": "Series" diff --git a/templates/schema/template.schema.json b/templates/schema/template.schema.json index 1c860c7..c1d997f 100644 --- a/templates/schema/template.schema.json +++ b/templates/schema/template.schema.json @@ -678,6 +678,31 @@ "title": "DeviceTemplate", "type": "object" }, + "EnergyDashboardSeries": { + "additionalProperties": false, + "description": "Series read from the WebSocket command ``energy/solar_forecast`` (Phase 4, forecasts only).\n\nThe command returns ``{config_entry_id: {\"wh_hours\": {iso_timestamp: Wh}}}`` for the forecast\nintegrations linked in the Energy dashboard (Forecast.Solar in Home Assistant core). The\ntemplate names no entity: the user binds the source to a config entry\n(``ForecastSourceConfig.config_entry_id``), or SunCore takes the only entry there is.\n``fields`` gives only ``value_unit`` (energy per period; no quantiles). The length of a\nperiod is inferred from the spacing of the timestamps; the template's ``resolution_min``\nis the expected length (used for a series of one point).", + "properties": { + "fields": { + "$ref": "#/$defs/SeriesFields" + }, + "missing_is_zero": { + "default": false, + "title": "Missing Is Zero", + "type": "boolean" + }, + "source": { + "const": "energy_solar_forecast", + "title": "Source", + "type": "string" + } + }, + "required": [ + "source", + "fields" + ], + "title": "EnergyDashboardSeries", + "type": "object" + }, "EntityMatcher": { "additionalProperties": false, "description": "Role matcher; every given criterion must match, ``domain`` is mandatory.\n\n``units`` lists acceptable ``unit_of_measurement`` strings (exact, case-sensitive);\n``unit_pattern`` is a regular expression matched with ``re.fullmatch`` on the stripped\nunit. ``translation_key`` / ``translation_key_pattern`` likewise. Exact and pattern\nforms exclude each other.", @@ -889,6 +914,17 @@ "title": "Capabilities", "type": "array" }, + "fallback_series": { + "anyOf": [ + { + "$ref": "#/$defs/ActionSeries" + }, + { + "type": "null" + } + ], + "default": null + }, "forecast_kind": { "enum": [ "pv", @@ -951,7 +987,8 @@ "discriminator": { "mapping": { "action": "#/$defs/ActionSeries", - "attribute": "#/$defs/AttributeSeries" + "attribute": "#/$defs/AttributeSeries", + "energy_solar_forecast": "#/$defs/EnergyDashboardSeries" }, "propertyName": "source" }, @@ -961,6 +998,9 @@ }, { "$ref": "#/$defs/ActionSeries" + }, + { + "$ref": "#/$defs/EnergyDashboardSeries" } ], "title": "Series" diff --git a/tests/api/conftest.py b/tests/api/conftest.py index 3efc871..29aecc5 100644 --- a/tests/api/conftest.py +++ b/tests/api/conftest.py @@ -51,6 +51,8 @@ from suncore.core.signals import Quality from suncore.core.site import EntityRef, GridConfig, PvArray, SiteConfig from suncore.core.wizard import CheckItem, PreviewItem, PreviewTooLargeError +from suncore.forecast.accuracy import accuracy_view +from suncore.forecast.views import ForecastAccuracyView, PvForecastView from suncore.storage import ( HistoryEntry, HistoryEntryNotFoundError, @@ -208,6 +210,16 @@ def __init__(self) -> None: self.proposal_list: list[Decision] = [] self.operation = OperationConfig() self.control_listeners: list[Callable[[], None]] = [] + self.pv_view = PvForecastView(generated_at=NOW, hours=48, sources=[], blended=None) + self.accuracy = accuracy_view([], {}) + self.pv_hours: list[int] = [] + + async def pv_forecast(self, hours: int) -> PvForecastView: + self.pv_hours.append(hours) + return self.pv_view + + async def forecast_accuracy(self) -> ForecastAccuracyView: + return self.accuracy async def health(self) -> HealthReport: self.health_calls += 1 diff --git a/tests/api/test_forecast_api.py b/tests/api/test_forecast_api.py new file mode 100644 index 0000000..6d318d6 --- /dev/null +++ b/tests/api/test_forecast_api.py @@ -0,0 +1,167 @@ +# SPDX-License-Identifier: Apache-2.0 +"""``GET /api/v1/forecast/pv`` and ``/forecast/accuracy`` (Phase 4 WP5; read only, no UI).""" + +from __future__ import annotations + +from suncore.api import create_app +from suncore.core.forecast import AccuracyRecord, HorizonBucket +from suncore.forecast.accuracy import StoredAccuracy, accuracy_view +from suncore.forecast.views import ( + BlendedView, + BlendWeightsView, + CalibrationView, + PvForecastView, + PvSeriesView, + PvSourceView, +) +from tests.api.conftest import NOW, FakeServices, client_for + + +def pv_view() -> PvForecastView: + series = PvSeriesView( + start=NOW, + step_s=1800, + p10_w=[None, 600.0], + p50_w=[None, 1000.0], + p90_w=[None, 1200.0], + ) + source = PvSourceView( + source_id="solcast.pv", + status="ok", + fetched_at=NOW, + age_s=12.5, + stale=False, + filled_slots=0, + series=series, + ) + never = PvSourceView( + source_id="forecast_solar.pv", + status="not_linked", + fetched_at=None, + age_s=None, + stale=True, + filled_slots=0, + series=None, + ) + blended = BlendedView( + method="single_source", + sources=["solcast.pv"], + start=NOW, + step_s=900, + p10_w=[None, 600.0], + p50_w=[None, 1000.0], + p90_w=[None, 1200.0], + uncalibrated_p50_w=[None, 1000.0], + calibrated_p50_w=[None, 900.0], + calibration_active=False, + stale=False, + excluded_sources=[], + weights=[ + BlendWeightsView( + horizon=HorizonBucket.H0_6, + basis="equal_no_history", + manual=False, + weights={"solcast.pv": 1.0}, + ) + ], + calibration=[ + CalibrationView( + source_id="solcast.pv", + active=False, + reason="collecting", + days_with_data=12, + min_history_days=30, + cells=40, + factor_min=0.85, + factor_max=0.95, + ) + ], + ) + return PvForecastView(generated_at=NOW, hours=24, sources=[source, never], blended=blended) + + +async def test_pv_forecast_shape_and_default_hours(services: FakeServices) -> None: + services.pv_view = pv_view() + async with client_for(create_app(services)) as client: + r = await client.get("/api/v1/forecast/pv") + r24 = await client.get("/api/v1/forecast/pv", params={"hours": 24}) + assert r.status_code == 200 + assert r24.status_code == 200 + assert services.pv_hours == [48, 24] + body = r.json() + assert [s["source_id"] for s in body["sources"]] == ["solcast.pv", "forecast_solar.pv"] + first = body["sources"][0] + assert first["series"]["p50_w"] == [None, 1000.0] + assert first["series"]["step_s"] == 1800 + assert first["stale"] is False + assert first["age_s"] == 12.5 + never = body["sources"][1] + assert never["series"] is None + assert never["fetched_at"] is None + assert never["stale"] is True + blend = body["blended"] + assert blend["method"] == "single_source" + assert blend["p50_w"] == [None, 1000.0] + assert blend["uncalibrated_p50_w"] == [None, 1000.0] + assert blend["calibrated_p50_w"] == [None, 900.0] + assert blend["calibration_active"] is False + assert blend["weights"] == [ + { + "horizon": "0_6h", + "basis": "equal_no_history", + "manual": False, + "weights": {"solcast.pv": 1.0}, + } + ] + assert blend["calibration"][0]["reason"] == "collecting" + assert blend["calibration"][0]["days_with_data"] == 12 + + +async def test_pv_forecast_without_sources(services: FakeServices) -> None: + async with client_for(create_app(services)) as client: + body = (await client.get("/api/v1/forecast/pv")).json() + assert body["sources"] == [] + assert body["blended"] is None + + +async def test_pv_forecast_hours_are_validated_with_the_standard_error( + services: FakeServices, +) -> None: + async with client_for(create_app(services)) as client: + for hours in (0, 97, "x"): + r = await client.get("/api/v1/forecast/pv", params={"hours": hours}) + assert r.status_code == 422 + assert services.pv_hours == [] + + +async def test_accuracy_shape(services: FakeServices) -> None: + record = AccuracyRecord( + source_id="solcast.pv", horizon=HorizonBucket.H6_24, mae_w=150.5, bias_w=-20.0, n=300 + ) + stored = {"solcast.pv": StoredAccuracy("solcast.pv", NOW, 14, (record,))} + services.accuracy = accuracy_view(["solcast.pv", "forecast_solar.pv"], stored) + async with client_for(create_app(services)) as client: + r = await client.get("/api/v1/forecast/accuracy") + body = r.json() + assert r.status_code == 200 + assert body["min_samples"] == 96 + solcast, other = body["sources"] + assert solcast["records"] == [ + {"source_id": "solcast.pv", "horizon": "6_24h", "mae_w": 150.5, "bias_w": -20.0, "n": 300} + ] + assert solcast["window_days"] == 14 + assert solcast["computed_at"] is not None + assert other["records"] == [] + assert other["computed_at"] is None + + +async def test_the_openapi_document_describes_both_routes(services: FakeServices) -> None: + schema = create_app(services).openapi() + for path in ("/api/v1/forecast/pv", "/api/v1/forecast/accuracy"): + assert "get" in schema["paths"][path] + description = schema["paths"]["/api/v1/forecast/pv"]["get"]["description"] + assert "blended" in description + assert "calibration" in description + accuracy = schema["paths"]["/api/v1/forecast/accuracy"]["get"]["description"] + assert "actual -" in accuracy + assert "blended" in accuracy diff --git a/tests/app/test_demo.py b/tests/app/test_demo.py index d461795..6339205 100644 --- a/tests/app/test_demo.py +++ b/tests/app/test_demo.py @@ -115,7 +115,16 @@ async def test_demo_live_feed_balance_closes(demo_runtime: Runtime) -> None: } ) """The demo starts with devices; the runtime subscribes to their entities too.""" -DEMO_SUBSCRIBED = frozenset(SIM_ENTITY_IDS) | DEMO_DEVICE_ENTITIES +DEMO_FORECAST_ENTITIES = frozenset( + { + "sensor.sim_solcast_forecast_today", + "sensor.sim_solcast_forecast_tomorrow", + "sensor.sim_open_meteo_energy_production_today", + "sensor.sim_open_meteo_energy_production_tomorrow", + } +) +"""The configured PV forecast sources' day sensors are subscribed too (Phase 4 WP5).""" +DEMO_SUBSCRIBED = frozenset(SIM_ENTITY_IDS) | DEMO_DEVICE_ENTITIES | DEMO_FORECAST_ENTITIES async def test_demo_starts_fully_configured(demo_runtime: Runtime) -> None: @@ -139,7 +148,11 @@ async def test_demo_starts_fully_configured(demo_runtime: Runtime) -> None: assert tariff.import_.price_entity == EntityRef(entity_id="sensor.sim_electricity_price") forecast = await demo_runtime.get_section("forecast") assert isinstance(forecast, ForecastConfig) - assert forecast.template_ids == ["solcast.pv"] + assert forecast.template_ids == [ + "solcast.pv", + "open_meteo_solar_forecast.pv", + "forecast_solar.pv", + ] operation = await demo_runtime.get_section("operation") assert isinstance(operation, OperationConfig) assert operation.mode is OperatingMode.SHADOW @@ -357,7 +370,7 @@ async def test_put_devices_resubscribes_and_shadows_new_ones(demo_runtime: Runti stored = await demo_runtime.put_section("devices", DevicesConfig(devices=[_switch("heater")])) assert isinstance(stored, DevicesConfig) assert await demo_runtime.get_section("operation") == OperationConfig() - assert conn.subscriptions[-1] == frozenset(SIM_ENTITY_IDS) | { + assert conn.subscriptions[-1] == frozenset(SIM_ENTITY_IDS) | DEMO_FORECAST_ENTITIES | { "switch.heater", "sensor.heater_power", } @@ -392,4 +405,4 @@ async def test_import_reloads_devices(demo_runtime: Runtime) -> None: exported["sections"]["devices"]["body"]["devices"] = [] await demo_runtime.import_config(exported) assert demo_runtime.devices == DevicesConfig() - assert conn.subscriptions[-1] == frozenset(SIM_ENTITY_IDS) + assert conn.subscriptions[-1] == frozenset(SIM_ENTITY_IDS) | DEMO_FORECAST_ENTITIES diff --git a/tests/app/test_pv_runtime.py b/tests/app/test_pv_runtime.py new file mode 100644 index 0000000..6cc3408 --- /dev/null +++ b/tests/app/test_pv_runtime.py @@ -0,0 +1,269 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The PV forecast service in the runtime: wiring, demo forecasts, accuracy job (Phase 4 WP5).""" + +from __future__ import annotations + +import asyncio +import itertools +from collections.abc import AsyncIterator, Callable +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from suncore.app.main import build_runtime +from suncore.app.runtime import Runtime +from suncore.core.config import ForecastConfig, ForecastSourceConfig +from suncore.core.forecast import HorizonBucket +from suncore.core.measurements import SITE_PV_W +from suncore.core.series import QuantileSeries, SeriesUnit, TimeSeries +from suncore.core.site import EntityRef +from suncore.ha import ConnectionState +from tests.app.conftest import RecordingConnection, demo_settings + +SCENARIO_DAY = datetime(2026, 6, 21, tzinfo=UTC) + + +async def until(predicate: Callable[[], bool], timeout_s: float = 10.0) -> None: + async with asyncio.timeout(timeout_s): + while not predicate(): # noqa: ASYNC110 - polling a plain predicate + await asyncio.sleep(0.01) + + +async def fast_sleep(delay: float) -> None: + """The refresh interval stays long; the debounce after an entity change is short.""" + await asyncio.sleep(0.01 if delay < 100 else delay) + + +@pytest.fixture +async def runtime(tmp_path: Path) -> AsyncIterator[Runtime]: + conn = RecordingConnection(speed=6000.0) + + def clock() -> datetime: + return conn.sim_time or SCENARIO_DAY + + rt = await build_runtime(demo_settings(tmp_path), env={}, connection=conn, clock=clock) + rt.control_sleep = fast_sleep + await rt._rebuild_pv_service() # the service takes the sleep it is built with + await rt.start() + try: + yield rt + finally: + await rt.stop() + + +async def test_the_demo_shows_forecasts_of_all_three_sources(runtime: Runtime) -> None: + assert runtime.pv_service is not None + assert runtime.pv_service.source_ids == [ + "solcast.pv", + "open_meteo_solar_forecast.pv", + "forecast_solar.pv", + ] + await until( + lambda: all( + (s := runtime.pv_service.snapshot(i)) is not None and s.series is not None # type: ignore[union-attr] + for i in runtime.pv_service.source_ids # type: ignore[union-attr] + ) + ) + view = await runtime.pv_forecast(48) + assert [s.status for s in view.sources] == ["ok", "ok", "ok"] + steps = {s.source_id: s.series.step_s for s in view.sources if s.series is not None} + assert steps == { + "solcast.pv": 1800, + "open_meteo_solar_forecast.pv": 900, + "forecast_solar.pv": 3600, + } + assert all(not s.stale for s in view.sources) + blended = view.blended + assert blended is not None + assert blended.sources == [ + "solcast.pv", + "open_meteo_solar_forecast.pv", + "forecast_solar.pv", + ] + assert blended.method == "weighted" + assert blended.step_s == 900 + assert {w.basis for w in blended.weights} == {"equal_no_history"} # no history yet + assert not blended.calibration_active + assert {c.reason for c in blended.calibration} == {"no_history"} + assert blended.calibrated_p50_w is None + assert blended.uncalibrated_p50_w == blended.p50_w + for lo, mid, hi in zip(blended.p10_w, blended.p50_w, blended.p90_w, strict=True): + if mid is not None: + assert lo is not None + assert hi is not None + assert lo <= mid <= hi + # the location of Home Assistant reaches the clear-sky model + await until(lambda: runtime._ha_location is not None) + assert runtime._blend_context().plant is not None + solcast = view.sources[0].series + assert solcast is not None + known = [v for v in solcast.p50_w if v is not None] + assert known + assert max(known) > 1000 + # P10 <= P50 <= P90 where the source has quantiles + for lo, mid, hi in zip(solcast.p10_w, solcast.p50_w, solcast.p90_w, strict=True): + if lo is not None and mid is not None and hi is not None: + assert lo <= mid <= hi + + +async def test_forecasts_are_stored_in_timeseries_db_at_most_hourly(runtime: Runtime) -> None: + store = runtime.timeseries + assert store is not None + assert runtime.pv_service is not None + await until(lambda: runtime.pv_service.snapshot("solcast.pv").last_stored_at is not None) # type: ignore[union-attr] + issues = await store.forecast_issues("solcast.pv", "pv") + assert issues + gaps = [b - a for a, b in itertools.pairwise(issues)] + assert all(g >= timedelta(minutes=15) for g in gaps) # never faster than the minimum gap + back = await store.read_forecast("solcast.pv", "pv", issues[0]) + assert back is not None + assert back.step == timedelta(minutes=30) + + +async def test_changing_the_forecast_section_rebuilds_the_service(runtime: Runtime) -> None: + conn = runtime.connection + assert isinstance(conn, RecordingConnection) + old = runtime.pv_service + only_open_meteo = ForecastConfig( + sources=[ + ForecastSourceConfig( + template_id="open_meteo_solar_forecast.pv", + entities={ + "energy_today": EntityRef( + entity_id="sensor.sim_open_meteo_energy_production_today" + ), + "energy_tomorrow": EntityRef( + entity_id="sensor.sim_open_meteo_energy_production_tomorrow" + ), + }, + ) + ] + ) + await runtime.put_section("forecast", only_open_meteo) + assert runtime.pv_service is not old + assert runtime.pv_service is not None + assert runtime.pv_service.source_ids == ["open_meteo_solar_forecast.pv"] + assert "sensor.sim_solcast_forecast_today" not in conn.subscriptions[-1] + await runtime.put_section("forecast", ForecastConfig()) + assert runtime.pv_service is None + view = await runtime.pv_forecast(24) + assert view.sources == [] + assert view.blended is None + accuracy = await runtime.forecast_accuracy() + assert accuracy.sources == [] + + +async def test_unusable_templates_are_skipped_not_fatal(runtime: Runtime) -> None: + config = ForecastConfig( + sources=[ + ForecastSourceConfig(template_id="no.such_template"), + ForecastSourceConfig(template_id="tibber.price"), # a tariff, not a forecast + ] + ) + await runtime.put_section("forecast", config) + assert runtime.pv_service is None + + +async def test_the_nightly_job_stores_accuracy_that_the_api_reads(runtime: Runtime) -> None: + store, learned = runtime.timeseries, runtime.learned + assert store is not None + assert learned is not None + now = runtime.clock() + day = (now - timedelta(days=1)).replace(hour=0, minute=0, second=0, microsecond=0) + quarters = 96 + await store.store_quarter_hours( + SITE_PV_W, + TimeSeries( + start=day, step=timedelta(minutes=15), values=(1000.0,) * quarters, unit=SeriesUnit.W + ), + ) + issued = day - timedelta(hours=6) + forecast = TimeSeries( + start=issued, step=timedelta(minutes=30), values=(700.0,) * 144, unit=SeriesUnit.W + ) + await store.store_forecast("solcast.pv", "pv", issued, QuantileSeries.deterministic(forecast)) + assert await runtime.maintain_timeseries() is True # starts the accuracy job in the background + assert runtime._accuracy_task is not None + await runtime._accuracy_task + accuracy = await runtime.forecast_accuracy() + by_id = {s.source_id: s for s in accuracy.sources} + assert set(by_id) == { + "solcast.pv", + "open_meteo_solar_forecast.pv", + "forecast_solar.pv", + "blended", + } + record = next(r for r in by_id["solcast.pv"].records if r.horizon is HorizonBucket.H6_24) + assert record.bias_w == 300.0 + assert record.mae_w == 300.0 + assert record.n > 0 + assert by_id["open_meteo_solar_forecast.pv"].records == [] + + +async def test_the_accuracy_job_never_raises(runtime: Runtime) -> None: + assert runtime.learned is not None + + async def boom(*_args: object, **_kw: object) -> None: + msg = "learned store gone" + raise OSError(msg) + + runtime.learned.put = boom # type: ignore[method-assign] + await runtime.update_forecast_accuracy() # logged, not raised + + +async def test_a_reconnect_wakes_the_pv_service(runtime: Runtime) -> None: + service = runtime.pv_service + assert service is not None + woken: list[bool] = [] + service.wake = lambda: woken.append(True) # type: ignore[method-assign] + runtime._on_state(ConnectionState.CONNECTED) + assert woken == [True] + + +async def test_stop_during_the_nightly_learning_releases_devices_first(tmp_path: Path) -> None: + """The nightly PV job may sit in a slow storage call when SunCore stops: the devices are + released first, the job is awaited after that, and the databases close last.""" + conn = RecordingConnection(speed=6000.0) + rt = await build_runtime( + demo_settings(tmp_path), env={}, connection=conn, clock=lambda: SCENARIO_DAY + ) + await rt.start() + assert rt.stack is not None + assert rt.timeseries is not None + assert rt.timeseries_db is not None + order: list[str] = [] + entered = asyncio.Event() + + async def slow_read(*_a: object, **_k: object) -> TimeSeries: + entered.set() + try: + await asyncio.Event().wait() + except asyncio.CancelledError: + await asyncio.sleep(0.05) # a cancelled statement's rollback takes a moment + order.append("job_ended") + raise + raise AssertionError # pragma: no cover + + rt.timeseries.read_series = slow_read # type: ignore[method-assign] + shutdown, ts_close, db_close = rt.stack.shutdown, rt.timeseries_db.close, rt.db.close + + async def record_shutdown() -> None: + order.append("release") + await shutdown() + + async def record_ts_close() -> None: + order.append("timeseries_closed") + await ts_close() + + async def record_db_close() -> None: + order.append("db_closed") + await db_close() + + rt.stack.shutdown = record_shutdown # type: ignore[method-assign] + rt.timeseries_db.close = record_ts_close # type: ignore[method-assign] + rt.db.close = record_db_close # type: ignore[method-assign] + rt._schedule_forecast_accuracy() + await asyncio.wait_for(entered.wait(), 5) + await rt.stop() + assert order == ["release", "job_ended", "timeseries_closed", "db_closed"] diff --git a/tests/forecast/__init__.py b/tests/forecast/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/forecast/conftest.py b/tests/forecast/conftest.py new file mode 100644 index 0000000..01a0534 --- /dev/null +++ b/tests/forecast/conftest.py @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Fixtures of the PV forecast tests: documented point shapes and a scripted reader. + +The shapes are the ones ``docs/research/integrations.md`` "PV forecasts" documents: + +* Solcast ``detailedForecast`` (research lines 36-47): entries ``{"period_start": , "pv_estimate": f, "pv_estimate10": f, "pv_estimate90": f}``, kW average power over + 30 minutes, one sensor per local day (``today``, ``tomorrow``, ``total_kwh_forecast_d3`` ...), + ``period_start`` in Home Assistant's time zone with the local offset; +* Solcast action ``query_forecast_data`` (lines 67-74): ``{"data": [same entries]}``; +* Forecast.Solar via ``energy/solar_forecast`` (lines 92-109): ``{"wh_hours": {iso: Wh}}``, + hourly, only a zero value at exactly 00:00:00 dropped; +* Open-Meteo Solar Forecast ``wh_period_15m`` (lines 133-144): ``{iso interval start: Wh}``, + 15 minutes, fixed UTC offset of the location. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from datetime import UTC, date, datetime, timedelta +from pathlib import Path +from typing import Any +from zoneinfo import ZoneInfo + +import pytest + +from suncore.templates import ForecastTemplate, TemplateCatalog, load_templates + +ROOT = Path(__file__).resolve().parents[2] +BERLIN = ZoneInfo("Europe/Berlin") + + +@pytest.fixture(scope="session") +def catalog() -> TemplateCatalog: + return load_templates(ROOT / "templates") + + +def forecast_template(catalog: TemplateCatalog, template_id: str) -> ForecastTemplate: + template = catalog.get(template_id) + assert isinstance(template, ForecastTemplate) + return template + + +def local_day_slots(day: date, step_min: int, tz: ZoneInfo = BERLIN) -> list[datetime]: + """Start of every slot of one local day, as aware datetimes in ``tz``. + + Walked in absolute time, so a DST day has 23 or 25 hours: 46 / 50 half-hour slots. + """ + begin = datetime.combine(day, datetime.min.time(), tzinfo=tz).astimezone(UTC) + end = datetime.combine(day + timedelta(days=1), datetime.min.time(), tzinfo=tz).astimezone(UTC) + slots = [] + moment = begin + while moment < end: + slots.append(moment.astimezone(tz)) + moment += timedelta(minutes=step_min) + return slots + + +def solcast_entries( + day: date, kw: Callable[[datetime], float] | None = None, tz: ZoneInfo = BERLIN +) -> list[dict[str, Any]]: + """``detailedForecast`` of one local day: P10 = 0.6 x P50, P90 = 1.2 x P50 (kW).""" + value = kw or (lambda s: 2.0 if 8 <= s.hour < 18 else 0.0) + return [ + { + "period_start": slot.isoformat(), + "pv_estimate": value(slot), + "pv_estimate10": round(value(slot) * 0.6, 6), + "pv_estimate90": round(value(slot) * 1.2, 6), + } + for slot in local_day_slots(day, 30, tz) + ] + + +class FakeReader: + """A scripted :class:`~suncore.forecast.pv.adapter.PvDataReader`.""" + + def __init__(self) -> None: + self.states: dict[str, Mapping[str, Any] | None] = {} + self.entries: dict[str, dict[str, float]] = {} + self.action_data: Any = {} + self.action_calls: list[tuple[str, str, dict[str, Any], float]] = [] + self.action_error: Exception | None = None + self.solar_calls = 0 + + def attributes(self, entity_id: str) -> Mapping[str, Any] | None: + return self.states.get(entity_id) + + async def solar_forecast(self) -> Mapping[str, Mapping[str, float]]: + self.solar_calls += 1 + return self.entries + + async def action_response( + self, domain: str, service: str, data: Mapping[str, Any], *, min_interval_s: float + ) -> Mapping[str, Any]: + self.action_calls.append((domain, service, dict(data), min_interval_s)) + if self.action_error is not None: + raise self.action_error + return self.action_data diff --git a/tests/forecast/test_accuracy.py b/tests/forecast/test_accuracy.py new file mode 100644 index 0000000..9e28d31 --- /dev/null +++ b/tests/forecast/test_accuracy.py @@ -0,0 +1,246 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Accuracy bookkeeping: MAE, bias (actual - forecast) and n per source and horizon bucket.""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +import pytest + +from suncore.core.forecast import ACCURACY_MIN_SAMPLES, AccuracyRecord, HorizonBucket +from suncore.core.measurements import SITE_PV_W +from suncore.core.series import QuantileSeries, SeriesUnit, TimeSeries +from suncore.forecast.accuracy import ( + NS_FORECAST_ACCURACY, + accuracy_view, + compute_accuracy, + load_accuracy, + no_clipping, + quarter_errors, + save_accuracy, + update_accuracy, +) +from suncore.storage import Database, LearnedParamStore +from suncore.storage.learned import NS_FORECAST_ACCURACY as STORAGE_NS +from suncore.storage.timeseries import TimeseriesStore, open_timeseries + +D = datetime(2026, 6, 21, 0, 0, tzinfo=UTC) # the day measured +NOW = D + timedelta(days=1) +Q = timedelta(minutes=15) + + +@pytest.fixture +async def store(tmp_path: Path) -> AsyncIterator[TimeseriesStore]: + db = await open_timeseries(tmp_path / "timeseries.db") + yield TimeseriesStore(db) + await db.close() + + +@pytest.fixture +async def learned(tmp_path: Path) -> AsyncIterator[LearnedParamStore]: + db = await Database.open(tmp_path / "suncore.db") + yield LearnedParamStore(db, clock=lambda: NOW) + await db.close() + + +def ts(values: list[float | None], start: datetime, step: timedelta = Q) -> TimeSeries: + return TimeSeries(start=start, step=step, values=tuple(values), unit=SeriesUnit.W) + + +async def measure(store: TimeseriesStore, values: list[float | None], start: datetime = D) -> None: + await store.store_quarter_hours(SITE_PV_W, ts(values, start)) + + +async def issue( + store: TimeseriesStore, + issued: datetime, + level: float, + *, + start: datetime | None = None, + hours: int = 72, + step: timedelta = Q, + source: str = "solcast.pv", +) -> None: + first = start or issued + count = int(timedelta(hours=hours) / step) + p50 = ts([level] * count, first, step) + await store.store_forecast(source, "pv", issued, QuantileSeries.deterministic(p50)) + + +def by_bucket(records: list[AccuracyRecord]) -> dict[HorizonBucket, AccuracyRecord]: + return {r.horizon: r for r in records} + + +# --- the maths ----------------------------------------------------------------------- + + +def test_error_is_actual_minus_forecast() -> None: + actual = ts([1000.0, 1000.0, 500.0], D) + forecast = ts([800.0, 1200.0, 500.0], D) + assert quarter_errors(forecast, actual) == [200.0, -200.0, 0.0] # + = under-forecast + + +def test_hourly_forecast_is_averaged_onto_quarter_hours_and_gaps_are_skipped() -> None: + actual = ts([100.0, 100.0, 100.0, 100.0, 100.0, 100.0], D) + forecast = ts([400.0, None], D, timedelta(hours=1)) # covers 4 quarters, then unknown + assert quarter_errors(forecast, actual) == [-300.0] * 4 + assert quarter_errors(ts([], D), actual) == [] + gappy = ts([100.0, None, 100.0, 100.0], D) + assert quarter_errors(ts([50.0] * 4, D), gappy) == [50.0, 50.0, 50.0] + + +async def test_under_forecast_gives_positive_bias_and_over_forecast_negative( + store: TimeseriesStore, +) -> None: + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(hours=6), 800.0, source="under") + await issue(store, D - timedelta(hours=6), 1300.0, source="over") + under = by_bucket(await compute_accuracy(store, "under", NOW)) + over = by_bucket(await compute_accuracy(store, "over", NOW)) + rec = under[HorizonBucket.H6_24] + assert (rec.bias_w, rec.mae_w, rec.horizon) == (200.0, 200.0, HorizonBucket.H6_24) + assert over[HorizonBucket.H6_24].bias_w == -300.0 + assert over[HorizonBucket.H6_24].mae_w == 300.0 + + +async def test_each_bucket_judges_the_issue_with_the_matching_lead(store: TimeseriesStore) -> None: + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(hours=6), 800.0) # D-1 18:00, 200 W too low + await issue(store, D + timedelta(hours=4), 1000.0) # D 04:00, perfect + records = by_bucket(await compute_accuracy(store, "solcast.pv", NOW)) + # 0-6 h: the newest issue not after the slot, lead < 6 h -> issue 2 for D 04:00-10:00 + near = records[HorizonBucket.H0_6] + assert (near.n, near.mae_w, near.bias_w) == (24, 0.0, 0.0) + # 6-24 h: issue 1 for D 00:00-10:00 (40 quarters, +200), issue 2 for D 10:00-24:00 (56, 0) + mid = records[HorizonBucket.H6_24] + assert mid.n == 96 + assert mid.bias_w == pytest.approx(200.0 * 40 / 96) + assert mid.mae_w == pytest.approx(200.0 * 40 / 96) + # 24-48 h: issue 1 for D 18:00-24:00 + far = records[HorizonBucket.H24_48] + assert (far.n, far.bias_w) == (24, 200.0) + + +async def test_an_issue_older_than_the_bucket_is_skipped_not_blamed(store: TimeseriesStore) -> None: + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(days=3), 100.0, hours=96) # 3 days stale: lead >= 48 h + assert await compute_accuracy(store, "solcast.pv", NOW) == [] + + +async def test_clipped_and_unmeasured_intervals_are_excluded(store: TimeseriesStore) -> None: + actual: list[float | None] = [1000.0] * 96 + for i in range(48, 56): # 12:00-14:00 the inverter clipped: measured far below the sun + actual[i] = 300.0 + for i in range(8): # no measurement 00:00-02:00 + actual[i] = None + await measure(store, actual) + await issue(store, D - timedelta(hours=6), 800.0) + plain = by_bucket(await compute_accuracy(store, "solcast.pv", NOW))[HorizonBucket.H6_24] + assert plain.n == 64 # D 00:00-18:00 = 72 quarters, 8 unmeasured ones never count as 0 + assert plain.bias_w != 200.0 + + def clipped(slot_start: datetime) -> bool: + return D + timedelta(hours=12) <= slot_start < D + timedelta(hours=14) + + filtered = by_bucket(await compute_accuracy(store, "solcast.pv", NOW, is_clipped=clipped))[ + HorizonBucket.H6_24 + ] + assert filtered.n == 56 + assert (filtered.bias_w, filtered.mae_w) == (200.0, 200.0) + assert no_clipping(D) is False # the default hook flags nothing (detection is part B) + + +async def test_half_hour_and_hourly_forecasts_are_compared_on_the_quarter_grid( + store: TimeseriesStore, +) -> None: + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(hours=6), 700.0, step=timedelta(minutes=30), source="half") + await issue(store, D - timedelta(hours=6), 900.0, step=timedelta(hours=1), source="hour") + half = by_bucket(await compute_accuracy(store, "half", NOW))[HorizonBucket.H6_24] + hour = by_bucket(await compute_accuracy(store, "hour", NOW))[HorizonBucket.H6_24] + assert (half.n, half.bias_w) == (72, 300.0) + assert (hour.n, hour.bias_w) == (72, 100.0) + + +async def test_a_source_without_data_has_no_records(store: TimeseriesStore) -> None: + await measure(store, [1000.0] * 96) + assert await compute_accuracy(store, "nothing.pv", NOW) == [] + empty = await compute_accuracy(store, "nothing.pv", NOW + timedelta(days=40)) + assert empty == [] + + +# --- persistence ------------------------------------------------------------------------- + + +async def test_records_round_trip_through_learned_param( + store: TimeseriesStore, learned: LearnedParamStore +) -> None: + assert NS_FORECAST_ACCURACY == STORAGE_NS + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(hours=6), 800.0) + results = await update_accuracy(store, learned, ["solcast.pv", "other.pv"], NOW) + assert set(results) == {"solcast.pv", "other.pv"} + assert results["other.pv"] == [] + stored = await load_accuracy(learned, ["solcast.pv", "other.pv"]) + assert stored["solcast.pv"].records == tuple(results["solcast.pv"]) + assert stored["solcast.pv"].computed_at == NOW + assert stored["solcast.pv"].window_days == 14 + view = accuracy_view(["solcast.pv", "other.pv", "new.pv"], stored) + assert view.min_samples == ACCURACY_MIN_SAMPLES + assert [s.source_id for s in view.sources] == ["solcast.pv", "other.pv", "new.pv"] + assert view.sources[0].records[0].bias_w == 200.0 + assert view.sources[2].computed_at is None + assert view.sources[2].records == [] + # a later night replaces the earlier result + await save_accuracy(learned, "solcast.pv", [], now=NOW + timedelta(days=1)) + assert (await load_accuracy(learned, ["solcast.pv"]))["solcast.pv"].records == () + + +async def test_unreadable_or_foreign_payloads_are_skipped(learned: LearnedParamStore) -> None: + await learned.put(NS_FORECAST_ACCURACY, "bad.pv", {"records": "nope"}, schema_version=1) + await learned.put(NS_FORECAST_ACCURACY, "future.pv", {"records": []}, schema_version=9) + assert await load_accuracy(learned) == {} + + +async def test_one_failing_source_does_not_stop_the_others(learned: LearnedParamStore) -> None: + class Broken: + async def read_series(self, *_args: object, **_kw: object) -> TimeSeries: + msg = "db gone" + raise OSError(msg) + + async def forecast_issues(self, *_args: object, **_kw: object) -> list[datetime]: + return [] + + async def read_forecast(self, *_args: object, **_kw: object) -> None: + return None + + assert await update_accuracy(Broken(), learned, ["a.pv", "b.pv"], NOW) == {} # type: ignore[arg-type] + + +async def test_every_issue_is_read_once_for_all_its_buckets(store: TimeseriesStore) -> None: + await measure(store, [1000.0] * 96) + await issue(store, D - timedelta(hours=3), 900.0) # 0-6 h, 6-24 h and 24-48 h of day D + calls: list[datetime] = [] + original = store.read_forecast + + async def counting(source_id: str, kind: Any, issued: datetime, window: Any = None) -> Any: + calls.append(issued) + return await original(source_id, kind, issued, window) + + store.read_forecast = counting # type: ignore[method-assign] + records = await compute_accuracy(store, "solcast.pv", NOW, window_days=1) + assert {r.horizon for r in records} == { + HorizonBucket.H0_6, + HorizonBucket.H6_24, + HorizonBucket.H24_48, + } + assert len(calls) == 1 # one read for all three buckets + + +def test_night_slots_where_both_are_zero_do_not_count() -> None: + actual = ts([0.0, 0.0, 500.0, 0.0], D) + forecast = ts([0.0, 100.0, 400.0, 0.0], D) + assert quarter_errors(forecast, actual) == [-100.0, 100.0] diff --git a/tests/forecast/test_pv_adapter.py b/tests/forecast/test_pv_adapter.py new file mode 100644 index 0000000..ff4146a --- /dev/null +++ b/tests/forecast/test_pv_adapter.py @@ -0,0 +1,344 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The template-driven PV forecast sources, from the documented fixtures (Phase 4 WP5).""" + +from __future__ import annotations + +import itertools +from datetime import UTC, date, datetime, timedelta +from typing import Any + +import pytest + +from suncore.core.forecast import ForecastSource +from suncore.core.series import SeriesUnit, TimeRange +from suncore.forecast.pv.adapter import FetchStatus, TemplateForecastSource +from suncore.templates import TemplateCatalog +from tests.forecast.conftest import ( + BERLIN, + FakeReader, + forecast_template, + local_day_slots, + solcast_entries, +) + +DAY = date(2026, 6, 21) +WINDOW = TimeRange( + start=datetime(2026, 6, 20, 22, 0, tzinfo=UTC), end=datetime(2026, 6, 24, 22, 0, tzinfo=UTC) +) +SOLCAST_ENTITIES = { + "forecast_today": "sensor.solcast_pv_forecast_forecast_today", + "forecast_tomorrow": "sensor.solcast_pv_forecast_forecast_tomorrow", + "forecast_d3": "sensor.solcast_pv_forecast_forecast_day_3", +} + + +def solcast( + catalog: TemplateCatalog, reader: FakeReader, entities: dict[str, str] | None = None +) -> TemplateForecastSource: + return TemplateForecastSource( + forecast_template(catalog, "solcast.pv"), + SOLCAST_ENTITIES if entities is None else entities, + reader, + ) + + +def day_state(day: date, kw: Any = None) -> dict[str, Any]: + return {"detailedForecast": solcast_entries(day, kw), "dataCorrect": True, "dayname": "x"} + + +async def test_the_source_satisfies_the_protocol(catalog: TemplateCatalog) -> None: + source = solcast(catalog, FakeReader()) + assert isinstance(source, ForecastSource) + assert source.source_id == "solcast.pv" + assert source.watched_entities == frozenset(SOLCAST_ENTITIES.values()) + + +async def test_solcast_joins_today_tomorrow_and_d3(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.states = { + SOLCAST_ENTITIES["forecast_today"]: day_state(DAY), + SOLCAST_ENTITIES["forecast_tomorrow"]: day_state(DAY + timedelta(days=1)), + SOLCAST_ENTITIES["forecast_d3"]: day_state(DAY + timedelta(days=2)), + } + source = solcast(catalog, reader) + series = await source.fetch(WINDOW) + assert series.unit is SeriesUnit.W + assert series.step == timedelta(minutes=30) + assert len(series) == 3 * 48 + assert series.start == datetime(2026, 6, 20, 22, tzinfo=UTC) + assert series.p50.values[24] == 2000.0 + assert series.p10.values[24] <= 2000.0 + assert source.last_report.status is FetchStatus.OK + assert source.last_report.entities_used == 3 + assert source.last_report.route == "attribute" + assert reader.action_calls == [] # the attributes were enough + + +async def test_a_missing_day_sensor_leaves_a_gap_and_is_reported(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.states = { + SOLCAST_ENTITIES["forecast_today"]: day_state(DAY), + SOLCAST_ENTITIES["forecast_tomorrow"]: None, # unavailable + SOLCAST_ENTITIES["forecast_d3"]: day_state(DAY + timedelta(days=2)), + } + source = solcast(catalog, reader) + series = await source.fetch(WINDOW) + assert len(series) == 3 * 48 + assert series.p50.values[48:96] == (None,) * 48 + assert source.last_report.status is FetchStatus.PARTIAL + assert source.last_report.entities_missing == (SOLCAST_ENTITIES["forecast_tomorrow"],) + + +async def test_one_irregular_sensor_is_dropped_the_others_kept(catalog: TemplateCatalog) -> None: + broken = day_state(DAY + timedelta(days=1)) + broken["detailedForecast"][3]["period_start"] = "2026-06-22T01:40:00+02:00" + reader = FakeReader() + reader.states = { + SOLCAST_ENTITIES["forecast_today"]: day_state(DAY), + SOLCAST_ENTITIES["forecast_tomorrow"]: broken, + } + source = solcast(catalog, reader) + series = await source.fetch(WINDOW) + assert len(series) == 48 + assert source.last_report.status is FetchStatus.PARTIAL + assert source.last_report.dropped_entities == (SOLCAST_ENTITIES["forecast_tomorrow"],) + + +async def test_window_crops_the_series(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.states = {SOLCAST_ENTITIES["forecast_today"]: day_state(DAY)} + source = solcast(catalog, reader) + noon = datetime(2026, 6, 21, 10, 0, tzinfo=UTC) # 12:00 local + series = await source.fetch(TimeRange(start=noon, end=noon + timedelta(hours=2))) + assert series.start == noon + assert len(series) == 4 + assert series.p50.values == (2000.0,) * 4 + + +async def test_attribute_off_falls_back_to_the_query_action(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.states = {SOLCAST_ENTITIES["forecast_today"]: {"dayname": "Sunday"}} # no attribute + reader.action_data = {"data": solcast_entries(DAY) + solcast_entries(DAY + timedelta(days=1))} + source = solcast(catalog, reader) + series = await source.fetch(WINDOW) + assert len(series) == 2 * 48 + assert series.p50.values[24] == 2000.0 + assert source.last_report.route == "action" + domain, service, data, interval = reader.action_calls[0] + assert (domain, service) == ("solcast_solar", "query_forecast_data") + assert data["start_date_time"] == WINDOW.start.isoformat() + assert datetime.fromisoformat(data["end_date_time"]) == WINDOW.start + timedelta(hours=96) + assert interval >= 60 + + +async def test_without_any_configured_entity_nothing_is_read(catalog: TemplateCatalog) -> None: + reader = FakeReader() + source = solcast(catalog, reader, entities={}) + series = await source.fetch(WINDOW) + assert len(series) == 0 + assert source.last_report.status is FetchStatus.NOT_CONFIGURED + assert reader.action_calls == [] # no opt-in, no action + + +async def test_action_errors_propagate_to_the_service(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.states = {SOLCAST_ENTITIES["forecast_today"]: None} + reader.action_error = RuntimeError("rate limited") + with pytest.raises(RuntimeError): + await solcast(catalog, reader).fetch(WINDOW) + + +async def test_all_unknown_data_is_no_data_not_a_series(catalog: TemplateCatalog) -> None: + reader = FakeReader() + entries = solcast_entries(DAY) + for entry in entries: + entry["pv_estimate"] = None + reader.states = {SOLCAST_ENTITIES["forecast_today"]: {"detailedForecast": entries}} + reader.action_data = {"data": []} + source = solcast(catalog, reader) + series = await source.fetch(WINDOW) + assert len(series) == 0 + assert source.last_report.status is FetchStatus.NO_DATA + + +# --- Open-Meteo Solar Forecast ----------------------------------------------------- + + +def open_meteo_day(day: date) -> dict[str, float]: + """``wh_period_15m`` of one local day, fixed offset of the location.""" + return { + slot.isoformat(): (125.0 if 8 <= slot.hour < 18 else 0.0) + for slot in local_day_slots(day, 15) + } + + +async def test_open_meteo_15_minute_wh_become_w(catalog: TemplateCatalog) -> None: + ids = {"energy_today": "sensor.today", "energy_tomorrow": "sensor.tomorrow"} + reader = FakeReader() + reader.states = { + "sensor.today": {"wh_period_15m": open_meteo_day(DAY), "watts": {}}, + "sensor.tomorrow": {"wh_period_15m": open_meteo_day(DAY + timedelta(days=1))}, + } + source = TemplateForecastSource( + forecast_template(catalog, "open_meteo_solar_forecast.pv"), ids, reader + ) + series = await source.fetch(WINDOW) + assert series.step == timedelta(minutes=15) + assert len(series) == 2 * 96 + assert series.p50.values[4 * 12] == 500.0 # 125 Wh in 15 min = 500 W + assert series.p10 == series.p50 == series.p90 # a single deterministic estimate + assert series.p50.total() == pytest.approx(2 * 40 * 125.0) # energy preserved + assert source.last_report.status is FetchStatus.OK + + +# --- Forecast.Solar through energy/solar_forecast ----------------------------------- + + +def forecast_solar_hours(day: date) -> dict[str, float]: + """Hourly Wh; only a zero hour at exactly 00:00:00 is dropped (``forecast_solar/energy.py`` + keeps ``val != 0 or (h, m, s) != (0, 0, 0)``).""" + hours = {} + for slot in local_day_slots(day, 60): + wh = 1000.0 if 8 <= slot.hour < 18 else 0.0 + if wh != 0 or (slot.hour, slot.minute, slot.second) != (0, 0, 0): + hours[slot.isoformat()] = wh + return hours + + +async def test_forecast_solar_reads_the_only_entry_and_fills_the_dropped_midnight( + catalog: TemplateCatalog, +) -> None: + reader = FakeReader() + reader.entries = { + "entry1": {**forecast_solar_hours(DAY), **forecast_solar_hours(DAY + timedelta(days=1))} + } + source = TemplateForecastSource(forecast_template(catalog, "forecast_solar.pv"), {}, reader) + series = await source.fetch(WINDOW) + assert series.step == timedelta(hours=1) + assert series.p10 == series.p90 == series.p50 + # day 1 starts at 01:00 (its 0 Wh midnight was dropped) and day 2 runs to its end + assert len(series) == 23 + 24 + assert series.p50.values[0] == 0.0 # 01:00, a listed zero hour + assert series.p50.values[22] == 0.0 # 23:00 + assert series.p50.values[23] == 0.0 # day 2's dropped midnight, filled between known hours + assert series.p50.values[9] == 1000.0 # 10:00: 1000 Wh in an hour = 1000 W + assert None not in series.p50.values + assert source.last_report.route == "energy" + + +async def test_forecast_solar_continuous_hours_with_only_midnight_missing( + catalog: TemplateCatalog, +) -> None: + """The shape ``forecast_solar/energy.py`` really produces: every hour, zeros included, only + the 0 Wh hour at 00:00:00 is absent.""" + hours = forecast_solar_hours(DAY + timedelta(days=1)) + stamps = sorted(datetime.fromisoformat(k) for k in hours) + assert len(stamps) == 23 + assert stamps[0].hour == 1 + assert all(b - a == timedelta(hours=1) for a, b in itertools.pairwise(stamps)) + previous = forecast_solar_hours(DAY) + reader = FakeReader() + reader.entries = {"entry1": {**previous, **hours}} + source = TemplateForecastSource(forecast_template(catalog, "forecast_solar.pv"), {}, reader) + series = await source.fetch(WINDOW) + midnight = datetime.combine(DAY + timedelta(days=1), datetime.min.time(), tzinfo=BERLIN) + index = (midnight - series.start) // timedelta(hours=1) + assert series.p50.values[index] == 0.0 + assert series.p50.total() == pytest.approx(2 * 10 * 1000.0) + + +async def test_forecast_solar_with_an_off_grid_key_still_delivers( + catalog: TemplateCatalog, +) -> None: + hours = forecast_solar_hours(DAY) + sunrise = datetime.combine(DAY, datetime.min.time(), tzinfo=BERLIN) + timedelta( + hours=7, minutes=41, seconds=13 + ) + reader = FakeReader() + reader.entries = {"entry1": {**hours, sunrise.isoformat(): 250.0}} + source = TemplateForecastSource(forecast_template(catalog, "forecast_solar.pv"), {}, reader) + series = await source.fetch(WINDOW) + assert series.step == timedelta(hours=1) + assert source.last_report.status is FetchStatus.OK + assert source.last_report.merged_points == 1 + assert series.p50.total() == pytest.approx(10 * 1000.0 + 250.0) + seven = (sunrise.replace(minute=0, second=0) - series.start) // timedelta(hours=1) + assert series.p50.values[seven] == 250.0 # the 07:00 hour was 0 Wh before + + +async def test_forecast_solar_with_several_entries_needs_a_choice(catalog: TemplateCatalog) -> None: + reader = FakeReader() + reader.entries = { + "entry_a": forecast_solar_hours(DAY), + "entry_b": {k: 2 * v for k, v in forecast_solar_hours(DAY).items()}, + } + template = forecast_template(catalog, "forecast_solar.pv") + ambiguous = TemplateForecastSource(template, {}, reader) + assert len(await ambiguous.fetch(WINDOW)) == 0 + assert ambiguous.last_report.status is FetchStatus.AMBIGUOUS_ENTRY + chosen = TemplateForecastSource(template, {}, reader, config_entry_id="entry_b") + series = await chosen.fetch(WINDOW) + assert series.p50.values[10] == 2000.0 + assert chosen.last_report.status is FetchStatus.OK + missing = TemplateForecastSource(template, {}, reader, config_entry_id="entry_c") + assert len(await missing.fetch(WINDOW)) == 0 + assert missing.last_report.status is FetchStatus.ENTRY_NOT_FOUND + + +async def test_forecast_solar_not_linked_in_the_energy_dashboard(catalog: TemplateCatalog) -> None: + reader = FakeReader() # energy/solar_forecast answers {} + source = TemplateForecastSource(forecast_template(catalog, "forecast_solar.pv"), {}, reader) + assert len(await source.fetch(WINDOW)) == 0 + assert source.last_report.status is FetchStatus.NOT_LINKED + + +async def test_half_hour_entries_are_recognised_by_their_spacing(catalog: TemplateCatalog) -> None: + # Solcast's energy platform: half-hour keys, Wh per 30 min; the period is inferred + reader = FakeReader() + reader.entries = {"e": {slot.isoformat(): 500.0 for slot in local_day_slots(DAY, 30)[16:24]}} + source = TemplateForecastSource(forecast_template(catalog, "forecast_solar.pv"), {}, reader) + series = await source.fetch(WINDOW) + assert series.step == timedelta(minutes=30) + assert series.p50.values == (1000.0,) * 8 # 500 Wh per 30 min = 1000 W + + +# --- generic attribute: list and mapping ----------------------------------------------- + + +async def test_generic_list_template(catalog: TemplateCatalog) -> None: + reader = FakeReader() + points = [ + {"start": slot.isoformat(), "power_w": 1500.0} for slot in local_day_slots(DAY, 30)[20:30] + ] + reader.states = {"sensor.my_forecast": {"forecast": points}} + source = TemplateForecastSource( + forecast_template(catalog, "generic.pv_forecast"), + {"forecast": "sensor.my_forecast"}, + reader, + ) + series = await source.fetch(WINDOW) + assert len(series) == 10 + assert series.p50.values == (1500.0,) * 10 + + +async def test_generic_mapping_template_with_a_list_path(catalog: TemplateCatalog) -> None: + base = forecast_template(catalog, "generic.pv_forecast") + series_spec = base.series.model_copy( + update={ + "shape": "mapping", + "list_path": "plan.watts", + "fields": base.series.fields.model_copy( + update={"timestamp_field": None, "value_field": None} + ), + } + ) + template = base.model_copy(update={"series": series_spec}) + reader = FakeReader() + stamps = local_day_slots(DAY, 30)[20:26] + reader.states = { + "sensor.x": {"forecast": {"plan": {"watts": {s.isoformat(): 800.0 for s in stamps}}}} + } + source = TemplateForecastSource(template, {"forecast": "sensor.x"}, reader) + series = await source.fetch(WINDOW) + assert series.p50.values == (800.0,) * 6 + assert stamps[0].tzinfo == BERLIN diff --git a/tests/forecast/test_pv_blend.py b/tests/forecast/test_pv_blend.py new file mode 100644 index 0000000..143bdf9 --- /dev/null +++ b/tests/forecast/test_pv_blend.py @@ -0,0 +1,237 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Blending of PV forecast sources: weights, minimum weight, manual override, quantiles.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from suncore.core.forecast import ACCURACY_MIN_SAMPLES, AccuracyRecord, HorizonBucket +from suncore.core.series import QuantileSeries, SeriesUnit, TimeSeries +from suncore.forecast.pv.blend import ( + MIN_WEIGHT, + WEIGHT_BUCKETS, + BlendInput, + blend, + blend_slot, + source_weights, + weight_bucket, +) + +NOW = datetime(2026, 6, 21, 4, 0, tzinfo=UTC) +Q = timedelta(minutes=15) +SLOTS = 48 * 4 + + +def series( + p50: list[float | None], + p10: list[float | None] | None = None, + p90: list[float | None] | None = None, + *, + start: datetime = NOW, + step: timedelta = Q, +) -> QuantileSeries: + def part(values: list[float | None]) -> TimeSeries: + return TimeSeries(start=start, step=step, values=tuple(values), unit=SeriesUnit.W) + + return QuantileSeries(p10=part(p10 or p50), p50=part(p50), p90=part(p90 or p50)) + + +def records(source: str, mae: float, n: int = ACCURACY_MIN_SAMPLES) -> list[AccuracyRecord]: + return [ + AccuracyRecord(source_id=source, horizon=b, mae_w=mae, bias_w=0.0, n=n) + for b in WEIGHT_BUCKETS + ] + + +def flat(level: float) -> QuantileSeries: + return series([level] * SLOTS) + + +# --- weights ------------------------------------------------------------------------- + + +def test_inverse_mae_weights_favour_the_better_source_but_keep_the_worse() -> None: + good = BlendInput("a", flat(1000), records=records("a", 100.0)) + bad = BlendInput("b", flat(1300), records=records("b", 400.0)) + weights = source_weights([good, bad], HorizonBucket.H0_6) + assert weights.auto_basis == "inverse_mae" + assert weights.weights["a"] > weights.weights["b"] >= MIN_WEIGHT + assert sum(weights.weights.values()) == pytest.approx(1.0) + # 1/100 : 1/400 -> skill 0.8 / 0.2, with the floor: 0.1 + 0.8*0.8, 0.1 + 0.8*0.2 + assert weights.weights["a"] == pytest.approx(0.74) + assert weights.weights["b"] == pytest.approx(0.26) + + +def test_equal_weights_until_every_auto_source_has_enough_history() -> None: + a = BlendInput("a", flat(1000), records=records("a", 100.0)) + b = BlendInput("b", flat(1000), records=records("b", 400.0, n=ACCURACY_MIN_SAMPLES - 1)) + weights = source_weights([a, b], HorizonBucket.H6_24) + assert weights.auto_basis == "equal_no_history" + assert weights.weights == {"a": pytest.approx(0.5), "b": pytest.approx(0.5)} + + +def test_a_hopeless_source_keeps_the_minimum_weight() -> None: + a = BlendInput("a", flat(1000), records=records("a", 1.0)) + b = BlendInput("b", flat(1000), records=records("b", 1e6)) + weights = source_weights([a, b], HorizonBucket.H24_48) + assert weights.weights["b"] == pytest.approx(MIN_WEIGHT, rel=1e-3) + + +def test_manual_weights_are_shares_and_auto_sources_split_the_rest() -> None: + manual = BlendInput("m", flat(1000), weight_mode="manual", manual_weight=0.6) + a = BlendInput("a", flat(1000)) + b = BlendInput("b", flat(1000)) + weights = source_weights([manual, a, b], HorizonBucket.H0_6) + assert weights.manual + assert weights.weights["m"] == pytest.approx(0.6) + assert weights.weights["a"] == pytest.approx(0.2) + assert weights.weights["b"] == pytest.approx(0.2) + + +def test_manual_weights_alone_are_normalised() -> None: + a = BlendInput("a", flat(1000), weight_mode="manual", manual_weight=0.3) + b = BlendInput("b", flat(1000), weight_mode="manual", manual_weight=0.9) + weights = source_weights([a, b], HorizonBucket.H0_6) + assert weights.auto_basis is None + assert weights.weights == {"a": pytest.approx(0.25), "b": pytest.approx(0.75)} + + +def test_manual_weights_summing_to_one_still_leave_the_auto_floor() -> None: + manual = BlendInput("m", flat(1000), weight_mode="manual", manual_weight=1.0) + auto = BlendInput("a", flat(1000)) + weights = source_weights([manual, auto], HorizonBucket.H0_6) + assert weights.weights["a"] == pytest.approx(MIN_WEIGHT) + assert weights.weights["m"] == pytest.approx(1 - MIN_WEIGHT) + + +@st.composite +def inputs(draw: st.DrawFn) -> list[BlendInput]: + count = draw(st.integers(1, 8)) + out = [] + for i in range(count): + manual = draw(st.booleans()) + mae = draw(st.one_of(st.none(), st.floats(0.0, 5000.0))) + n = draw(st.integers(0, 500)) + out.append( + BlendInput( + f"s{i}", + flat(1000), + weight_mode="manual" if manual else "auto", + manual_weight=draw(st.floats(0.01, 1.0)) if manual else None, + records=[] if mae is None else records(f"s{i}", mae, n), + ) + ) + return out + + +@given(inputs(), st.sampled_from(WEIGHT_BUCKETS)) +def test_weights_sum_to_one_and_auto_sources_keep_the_floor( + items: list[BlendInput], bucket: HorizonBucket +) -> None: + weights = source_weights(items, bucket).weights + assert sum(weights.values()) == pytest.approx(1.0) + floor = min(MIN_WEIGHT, 0.5 / len(items)) + for item in items: + assert weights[item.source_id] > 0 + if not item.is_manual: + assert weights[item.source_id] >= floor - 1e-12 + + +def test_weight_bucket_uses_24_48_beyond_48_hours() -> None: + assert weight_bucket(timedelta(hours=50)) is HorizonBucket.H24_48 + assert weight_bucket(timedelta(hours=1)) is HorizonBucket.H0_6 + + +# --- quantiles -------------------------------------------------------------------------- + + +quantile_triples = st.tuples( + st.floats(0.0, 10_000.0), st.floats(0.0, 5_000.0), st.floats(0.0, 5_000.0) +).map(lambda t: (max(0.0, t[0] - t[1]), t[0], t[0] + t[2])) + + +@given( + st.lists( + st.tuples(st.floats(0.01, 1.0), quantile_triples, st.booleans()), min_size=1, max_size=8 + ) +) +def test_blended_quantiles_are_ordered( + rows: list[tuple[float, tuple[float, float, float], bool]], +) -> None: + values = [(w, lo if q else mid, mid, hi if q else mid) for w, (lo, mid, hi), q in rows] + result = blend_slot(values) + assert result is not None + p10, p50, p90 = result + assert 0.0 <= p10 <= p50 <= p90 + mids = [v[2] for v in values] + assert min(mids) - 1e-6 <= p50 <= max(mids) + 1e-6 + + +@settings(max_examples=50) +@given(st.lists(quantile_triples, min_size=1, max_size=24)) +def test_a_single_source_passes_through_unchanged( + triples: list[tuple[float, float, float]], +) -> None: + source = series([t[1] for t in triples], [t[0] for t in triples], [t[2] for t in triples]) + result = blend([BlendInput("only", source)], NOW, horizon=Q * len(triples)) + assert result is not None + assert result.method == "single_source" + assert result.series == source + + +def test_agreeing_sources_keep_the_band_and_disagreeing_ones_widen_it() -> None: + agree = blend_slot([(0.5, 800.0, 1000.0, 1100.0), (0.5, 800.0, 1000.0, 1100.0)]) + assert agree == pytest.approx((800.0, 1000.0, 1100.0)) + apart = blend_slot([(0.5, 800.0, 1000.0, 1100.0), (0.5, 1000.0, 1200.0, 1300.0)]) + assert apart is not None + assert apart[1] == pytest.approx(1100.0) + assert apart[1] - apart[0] > 200.0 + assert apart[2] - apart[1] > 100.0 + + +def test_sources_without_quantiles_take_the_band_from_those_that_have_them() -> None: + result = blend_slot([(0.5, 600.0, 1000.0, 1200.0), (0.5, 1000.0, 1000.0, 1000.0)]) + assert result == pytest.approx((600.0, 1000.0, 1200.0)) + + +def test_without_any_quantiles_only_the_disagreement_makes_a_band() -> None: + result = blend_slot([(0.5, 1000.0, 1000.0, 1000.0), (0.5, 1000.0, 1000.0, 1000.0)]) + assert result == pytest.approx((1000.0, 1000.0, 1000.0)) + + +def test_weights_are_renormalised_over_the_sources_present_in_a_slot() -> None: + a = BlendInput("a", series([1000.0] * SLOTS), records=records("a", 100.0)) + b_values: list[float | None] = [None] * (SLOTS // 2) + [2000.0] * (SLOTS // 2) + b = BlendInput("b", series(b_values), records=records("b", 100.0)) + result = blend([a, b], NOW) + assert result is not None + assert result.series.p50.values[0] == pytest.approx(1000.0) + assert result.series.p50.values[-1] == pytest.approx(1500.0) + + +def test_the_blend_lies_on_the_planning_grid_for_48_hours() -> None: + half_hourly = series([500.0] * 96, step=timedelta(minutes=30), start=NOW - timedelta(hours=1)) + now = NOW + timedelta(minutes=7) + result = blend([BlendInput("a", half_hourly), BlendInput("b", flat(700))], now) + assert result is not None + assert result.series.step == Q + assert result.series.start == NOW + assert len(result.series) == SLOTS + 1 + assert result.series.p50.values[0] == pytest.approx(600.0) + + +def test_a_source_biased_30_percent_gets_the_lower_weight() -> None: + truth = 1000.0 + unbiased = BlendInput("a", flat(truth), records=records("a", 60.0)) + biased = BlendInput("b", flat(truth * 1.3), records=records("b", 300.0)) + result = blend([unbiased, biased], NOW) + assert result is not None + first = result.weights[0].weights + assert first["a"] > 3 * first["b"] + blended = result.series.p50.values[0] + assert blended is not None + assert abs(blended - truth) < abs(truth * 1.3 - truth) / 3 diff --git a/tests/forecast/test_pv_calibration.py b/tests/forecast/test_pv_calibration.py new file mode 100644 index 0000000..70cf3a1 --- /dev/null +++ b/tests/forecast/test_pv_calibration.py @@ -0,0 +1,340 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Bias calibration: clear-sky classes, bounded shrinking factors, activation, clipping.""" + +from __future__ import annotations + +from datetime import UTC, date, datetime, time, timedelta +from typing import Any +from zoneinfo import ZoneInfo + +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from suncore.core.series import QuantileSeries, SeriesUnit, TimeSeries +from suncore.core.site import PvArray +from suncore.forecast.pv.calibration import ( + FACTOR_BOUNDS, + NS_FORECAST_BIAS, + BiasTable, + CellStats, + calibrate, + calibration_state, + cell_factor, + decode, + encode, + learn_day, + load_bias, + new_table, + save_bias, +) +from suncore.forecast.pv.clearsky import ( + ArrayGeometry, + PlantGeometry, + WeatherClass, + classify_slots, + clear_sky_slot_w, + peak_reference, + plant_geometry, +) +from suncore.storage.learned import NS_FORECAST_BIAS as STORAGE_NS + +BERLIN = ZoneInfo("Europe/Berlin") +Q = timedelta(minutes=15) +SOUTH = PlantGeometry(48.0, 11.0, (ArrayGeometry(8.0, 180.0, 30.0),)) +EAST_WEST = PlantGeometry( + 48.0, 11.0, (ArrayGeometry(4.0, 90.0, 20.0), ArrayGeometry(4.0, 270.0, 20.0)) +) + + +def day_start(day: date) -> datetime: + return datetime.combine(day, time(), tzinfo=BERLIN).astimezone(UTC) + + +def day_series(day: date, fn: Any) -> TimeSeries: + start = day_start(day) + end = day_start(day + timedelta(days=1)) + count = (end - start) // Q + values = tuple(fn(start + Q * i) for i in range(count)) + return TimeSeries(start=start, step=Q, values=values, unit=SeriesUnit.W) + + +def clear(plant: PlantGeometry, kt: float = 0.85) -> Any: + return lambda begin: kt * clear_sky_slot_w(plant, begin, Q) + + +# --- clear-sky classes ------------------------------------------------------------------ + + +def test_plant_geometry_needs_location_and_every_kwp() -> None: + arrays = [PvArray(name="S", kwp=8.0), PvArray(name="N", kwp=2.0, azimuth_deg=0, tilt_deg=10)] + plant = plant_geometry(48.0, 11.0, arrays) + assert plant is not None + assert plant.arrays[0].azimuth_deg == 180.0 + assert plant.arrays[0].tilt_deg == 30.0 + assert plant_geometry(None, 11.0, arrays) is None + assert plant_geometry(48.0, 11.0, [PvArray(name="x")]) is None + assert plant_geometry(48.0, 11.0, []) is None + southern = plant_geometry(-33.0, 151.0, [PvArray(name="S", kwp=5.0)]) + assert southern is not None + assert southern.arrays[0].azimuth_deg == 0.0 + + +@pytest.mark.parametrize("plant", [SOUTH, EAST_WEST], ids=["south", "east_west"]) +def test_a_clear_forecast_is_clear_all_day_also_on_an_east_west_roof(plant: PlantGeometry) -> None: + day = date(2026, 6, 21) + classes = classify_slots(day_series(day, clear(plant)), plant=plant, tz=BERLIN) + daytime = [c for c in classes if c is not None] + assert len(daytime) > 50 + assert set(daytime) == {WeatherClass.CLEAR} + + +def test_an_east_west_profile_is_not_clear_against_a_south_reference() -> None: + """The class depends on the geometry: a south model reads an east-west morning as clear + sky only because the morning is strong - consistent, not necessarily equal.""" + day = date(2026, 6, 21) + forecast = day_series(day, clear(EAST_WEST, 0.85)) + south = classify_slots(forecast, plant=SOUTH, tz=BERLIN) + ew = classify_slots(forecast, plant=EAST_WEST, tz=BERLIN) + assert south != ew + + +def test_overcast_and_mixed_classes() -> None: + day = date(2026, 6, 21) + assert { + c for c in classify_slots(day_series(day, clear(SOUTH, 0.2)), plant=SOUTH, tz=BERLIN) if c + } == {WeatherClass.OVERCAST} + assert { + c for c in classify_slots(day_series(day, clear(SOUTH, 0.5)), plant=SOUTH, tz=BERLIN) if c + } == {WeatherClass.MIXED} + + +def test_without_a_location_the_daily_peak_is_the_reference() -> None: + day = date(2026, 6, 21) + sunny = day_series(day, clear(SOUTH, 0.85)) + dull = day_series(day + timedelta(days=1), clear(SOUTH, 0.2)) + both = TimeSeries( + start=sunny.start, step=Q, values=sunny.values + dull.values, unit=SeriesUnit.W + ) + classes = classify_slots(both, plant=None, tz=BERLIN) + first = {c for c in classes[: len(sunny)] if c} + second = {c for c in classes[len(sunny) :] if c} + assert first == {WeatherClass.CLEAR} + assert second == {WeatherClass.OVERCAST} + # a remembered peak makes a dull day alone overcast too + assert {c for c in classify_slots(dull, plant=None, tz=BERLIN, peak_reference_w=7000) if c} == { + WeatherClass.OVERCAST + } + + +# --- factors ---------------------------------------------------------------------------- + + +@given(st.floats(0.0, 1e6), st.floats(1e-3, 1e6), st.floats(0.0, 1e5), st.floats(0.0, 1e5)) +def test_factors_are_bounded_and_shrink_monotonically( + actual: float, forecast: float, n: float, more: float +) -> None: + few = cell_factor(CellStats(n=n, forecast_wh=forecast, actual_wh=actual)) + many = cell_factor(CellStats(n=n + more, forecast_wh=forecast, actual_wh=actual)) + lo, hi = FACTOR_BOUNDS + assert lo <= few <= hi + assert lo <= many <= hi + assert abs(many - 1.0) >= abs(few - 1.0) - 1e-12 # more samples never pull it nearer to 1 + raw = min(hi, max(lo, actual / forecast)) + assert min(1.0, raw) - 1e-12 <= few <= max(1.0, raw) + 1e-12 + + +def test_no_samples_means_factor_one() -> None: + assert cell_factor(None) == 1.0 + assert cell_factor(CellStats()) == 1.0 + + +# --- learning --------------------------------------------------------------------------- + + +def learn_days( + table: BiasTable, + days: int, + *, + plant: PlantGeometry | None, + forecast_kt: float = 0.85, + truth_kt: float = 0.85 / 1.3, + first: date = date(2026, 6, 1), + clipped: Any = None, +) -> BiasTable: + for i in range(days): + day = first + timedelta(days=i) + geometry = plant or SOUTH + forecast = day_series(day, clear(geometry, forecast_kt)) + actual = day_series(day, clear(geometry, truth_kt)) + table = learn_day( + table, + forecast, + actual, + plant=plant, + tz=BERLIN, + day_end=day_start(day + timedelta(days=1)), + is_clipped=clipped or (lambda _s: False), + ) + return table + + +def test_a_source_30_percent_too_high_is_corrected_after_30_days_only() -> None: + table = learn_days(new_table("b", SOUTH), 29, plant=SOUTH) + state = calibration_state("b", table, plant=SOUTH, min_history_days=30) + assert not state.active + assert state.reason == "collecting" + assert state.days_with_data == 29 + assert state.factor_max is not None + assert state.factor_max < 0.9 + table = learn_days(table, 1, plant=SOUTH, first=date(2026, 6, 30)) + state = calibration_state("b", table, plant=SOUTH, min_history_days=30) + assert state.active + assert state.reason == "active" + + day = date(2026, 7, 1) + forecast = day_series(day, clear(SOUTH, 0.85)) + truth = day_series(day, clear(SOUTH, 0.85 / 1.3)) + fixed = calibrate(QuantileSeries.deterministic(forecast), table, plant=SOUTH, tz=BERLIN) + raw_error = sum( + abs((f or 0) - (t or 0)) for f, t in zip(forecast.values, truth.values, strict=False) + ) + fixed_error = sum( + abs((f or 0) - (t or 0)) for f, t in zip(fixed.p50.values, truth.values, strict=False) + ) + assert fixed_error < 0.25 * raw_error + + +def test_an_east_west_roof_forecast_as_south_learns_hourly_factors() -> None: + """A source that models the roof as south-facing is too low in the morning and evening and + too high at noon; the hour-of-day cells learn exactly that.""" + table = new_table("s", EAST_WEST) + for i in range(30): + day = date(2026, 6, 1) + timedelta(days=i) + # the source scales a south profile to the east-west plant's daily energy + ew = day_series(day, clear(EAST_WEST, 0.85)) + south = day_series(day, clear(SOUTH, 0.85)) + scale = sum(v or 0 for v in ew.values) / sum(v or 0 for v in south.values) + forecast = TimeSeries( + start=south.start, + step=Q, + values=tuple((v or 0) * scale for v in south.values), + unit=SeriesUnit.W, + ) + table = learn_day( + table, + forecast, + ew, + plant=EAST_WEST, + tz=BERLIN, + day_end=day_start(day + timedelta(days=1)), + ) + morning = [c for k, c in table.cells.items() if k.startswith("08.jja")] + noon = [c for k, c in table.cells.items() if k.startswith("13.jja")] + assert morning + assert noon + assert all(cell_factor(c) > 1.05 for c in morning) + assert all(cell_factor(c) < 0.97 for c in noon) + + +def test_clipped_slots_are_not_learnt() -> None: + everything = learn_days(new_table("b", SOUTH), 3, plant=SOUTH) + nothing = learn_days(new_table("b", SOUTH), 3, plant=SOUTH, clipped=lambda _s: True) + assert everything.samples > 0 + assert nothing.samples == 0 + assert nothing.days_with_data == 0 + assert nothing.learned_until == everything.learned_until + + +@given(st.integers(1, 40), st.integers(1, 60)) +def test_inactive_before_min_history_days(days: int, minimum: int) -> None: + table = BiasTable( + "s", "clear_sky", {"12.jja.clear": CellStats(50, 1000, 700)}, days_with_data=days + ) + state = calibration_state("s", table, plant=SOUTH, min_history_days=minimum) + assert state.active is (days >= minimum) + + +def test_an_inactive_classifier_is_never_applied_and_never_learnt_over() -> None: + table = learn_days(new_table("b", SOUTH), 30, plant=SOUTH) + state = calibration_state("b", table, plant=None, min_history_days=30) + assert state.reason == "classifier_changed" + assert not state.active + kept = learn_days(table, 2, plant=None, first=date(2026, 7, 1)) + assert kept == table # a night without the location keeps the table + reset = learn_days(new_table("b", None), 2, plant=None) + assert reset.mode == "daily_peak" + upgraded = learn_days(reset, 1, plant=SOUTH, first=date(2026, 7, 1)) + assert upgraded.mode == "clear_sky" + assert upgraded.days_with_data == 1 + + +def test_calibrate_keeps_quantile_order_and_unknown_slots() -> None: + table = learn_days(new_table("b", SOUTH), 30, plant=SOUTH) + day = date(2026, 7, 2) + p50 = day_series(day, clear(SOUTH, 0.85)) + values = list(p50.values) + values[50] = None + p50 = TimeSeries(start=p50.start, step=Q, values=tuple(values), unit=SeriesUnit.W) + p10 = TimeSeries( + start=p50.start, + step=Q, + values=tuple(None if v is None else 0.7 * v for v in values), + unit=SeriesUnit.W, + ) + out = calibrate(QuantileSeries(p10=p10, p50=p50, p90=p50), table, plant=SOUTH, tz=BERLIN) + assert out.p50.values[50] is None + assert out.p50.values[52] is not None + assert values[52] is not None + assert out.p50.values[52] < values[52] + + +# --- persistence ------------------------------------------------------------------------ + + +class FakeLearned: + def __init__(self) -> None: + self.rows: dict[tuple[str, str], tuple[dict[str, Any], int]] = {} + + async def put(self, namespace: str, key: str, payload: Any, *, schema_version: int = 1) -> None: + self.rows[(namespace, key)] = (dict(payload), schema_version) + + async def in_namespace(self, namespace: str) -> list[Any]: + class Row: + def __init__(self, key: str, payload: dict[str, Any], version: int) -> None: + self.key, self.payload, self.schema_version = key, payload, version + + return [Row(k, p, v) for (ns, k), (p, v) in self.rows.items() if ns == namespace] + + +async def test_tables_round_trip_through_learned_params() -> None: + assert NS_FORECAST_BIAS == STORAGE_NS + table = learn_days(new_table("solcast.pv", SOUTH), 2, plant=SOUTH) + learned = FakeLearned() + await save_bias(learned, table) + loaded = await load_bias(learned) + assert loaded["solcast.pv"] == table + + +def test_unreadable_payloads_are_skipped() -> None: + assert decode("s", {"mode": "nope"}, 1) is None + assert decode("s", encode(new_table("s", SOUTH)), 2) is None + assert decode("s", {"mode": "clear_sky", "cells": [], "days_with_data": 1}, 1) is None + + +def test_one_absurd_forecast_peak_does_not_move_the_reference() -> None: + peaks = [5000.0 + 10 * i for i in range(40)] + [50_000.0] + table = BiasTable("s", "daily_peak", daily_peaks=tuple(peaks)) + reference = table.peak_reference_w + assert reference is not None + assert 5000.0 < reference < 5400.0 + assert peak_reference([]) is None + + +def test_daily_peaks_are_kept_for_a_year_and_survive_storage() -> None: + table = learn_days(new_table("b", None), 3, plant=None) + assert len(table.daily_peaks) == 3 + assert decode("b", encode(table), 1) == table + many = BiasTable("b", "daily_peak", daily_peaks=tuple(float(i + 1) for i in range(400))) + grown = learn_days(many, 1, plant=None) + assert len(grown.daily_peaks) == 365 diff --git a/tests/forecast/test_pv_clipping.py b/tests/forecast/test_pv_clipping.py new file mode 100644 index 0000000..e0c2edf --- /dev/null +++ b/tests/forecast/test_pv_clipping.py @@ -0,0 +1,153 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Clipping detection: export-limit plateau, inverter plateau from the data, natural peaks.""" + +from __future__ import annotations + +from datetime import UTC, date, datetime, time, timedelta +from typing import Any +from zoneinfo import ZoneInfo + +from suncore.core.series import SeriesUnit, TimeSeries +from suncore.core.site import EntityRef, GridConfig, PvArray, SiteConfig +from suncore.forecast.accuracy import quarter_errors +from suncore.forecast.pv.clearsky import ArrayGeometry, PlantGeometry, clear_sky_slot_w +from suncore.forecast.pv.clipping import ( + clip_filter, + clipped_slots, + export_limit_w, + plateau_cap_w, +) + +BERLIN = ZoneInfo("Europe/Berlin") +Q = timedelta(minutes=15) +PLANT = PlantGeometry(48.0, 11.0, (ArrayGeometry(10.0, 180.0, 30.0),)) + + +def days_series(first: date, days: int, fn: Any) -> TimeSeries: + start = datetime.combine(first, time(), tzinfo=BERLIN).astimezone(UTC) + count = days * 96 + return TimeSeries( + start=start, + step=Q, + values=tuple(fn(start + Q * i) for i in range(count)), + unit=SeriesUnit.W, + ) + + +def sun(kt: float = 0.8) -> Any: + return lambda begin: kt * clear_sky_slot_w(PLANT, begin, Q) + + +def test_export_limit_from_watts_or_percent_of_kwp() -> None: + meter = EntityRef(entity_id="sensor.grid") + watts = SiteConfig(grid=GridConfig(power=meter, export_limit_w=5000)) + assert export_limit_w(watts) == 5000 + pct = SiteConfig( + grid=GridConfig(power=meter, export_limit_pct_kwp=70), + pv=[PvArray(name="a", kwp=6.0), PvArray(name="b", kwp=4.0)], + ) + assert export_limit_w(pct) == 7000 + assert export_limit_w(SiteConfig(grid=GridConfig(power=meter))) is None + assert export_limit_w(None) is None + + +def test_a_plateau_at_the_export_limit_is_flagged() -> None: + limit = 4000.0 + load = 500.0 + first = date(2026, 6, 21) + potential = days_series(first, 1, sun()) + pv = TimeSeries( + start=potential.start, + step=Q, + values=tuple(min(v or 0.0, limit + load) for v in potential.values), + unit=SeriesUnit.W, + ) + grid = TimeSeries( + start=pv.start, + step=Q, + values=tuple(load - (v or 0.0) for v in pv.values), + unit=SeriesUnit.W, + ) + flagged = clipped_slots( + pv, tz=BERLIN, grid=grid, export_limit=limit, reference=potential, cap_w=None + ) + expected = { + b + for (b, _, p), (_, _, f) in zip(pv.slots(), potential.slots(), strict=True) + if f is not None and p is not None and f > p * 1.02 + } + assert flagged + assert flagged == expected + # without the forecast condition the flat export alone flags the same slots + assert clipped_slots(pv, tz=BERLIN, grid=grid, export_limit=limit) >= flagged + + +def test_export_at_the_limit_is_not_clipping_when_the_forecast_was_not_higher() -> None: + limit = 4000.0 + pv = days_series(date(2026, 6, 21), 1, lambda _b: 4500.0) + grid = days_series(date(2026, 6, 21), 1, lambda _b: -4000.0) + assert not clipped_slots(pv, tz=BERLIN, grid=grid, export_limit=limit, reference=pv) + + +def test_the_inverter_cap_is_derived_from_repeated_plateaus() -> None: + cap = 6000.0 + potential = days_series(date(2026, 6, 20), 3, sun(0.9)) + pv = TimeSeries( + start=potential.start, + step=Q, + values=tuple(min(v or 0.0, cap) for v in potential.values), + unit=SeriesUnit.W, + ) + assert plateau_cap_w(pv, BERLIN) == cap + flagged = clipped_slots(pv, tz=BERLIN, reference=potential) + assert flagged + assert all(pv.values[(b - pv.start) // Q] == cap for b in flagged) + + +def test_a_natural_peak_is_no_cap() -> None: + pv = days_series(date(2026, 6, 20), 3, sun(0.8)) + assert plateau_cap_w(pv, BERLIN) is None + assert not clipped_slots(pv, tz=BERLIN, reference=pv) + + +def test_one_clipped_day_is_not_enough_for_a_cap() -> None: + cap = 6000.0 + potential = days_series(date(2026, 6, 20), 3, sun(0.9)) + values = list(potential.values) + values[:96] = [min(v or 0.0, cap) for v in values[:96]] + values[96:] = [0.5 * (v or 0.0) for v in values[96:]] # two dull days + pv = TimeSeries(start=potential.start, step=Q, values=tuple(values), unit=SeriesUnit.W) + assert plateau_cap_w(pv, BERLIN) is None + + +def test_clipped_slots_are_excluded_from_the_accuracy() -> None: + limit = 4000.0 + potential = days_series(date(2026, 6, 21), 1, sun()) + pv = TimeSeries( + start=potential.start, + step=Q, + values=tuple(min(v or 0.0, limit) for v in potential.values), + unit=SeriesUnit.W, + ) + grid = TimeSeries( + start=pv.start, step=Q, values=tuple(-(v or 0.0) for v in pv.values), unit=SeriesUnit.W + ) + flagged = clipped_slots(pv, tz=BERLIN, grid=grid, export_limit=limit, reference=potential) + all_errors = quarter_errors(potential, pv) + kept = quarter_errors(potential, pv, is_clipped=clip_filter(flagged)) + assert len(kept) == len(all_errors) - len(flagged) + assert min(all_errors) < -100.0 + assert min(kept) > -1e-6 # the forecast was exact wherever nothing was clipped + + +def test_a_zero_export_limit_flags_nothing_by_the_export_rule() -> None: + """Documented decision: grid ~ 0 W is the normal state of a zero-export plant.""" + potential = days_series(date(2026, 6, 21), 1, sun()) + pv = TimeSeries( + start=potential.start, + step=Q, + values=tuple(0.6 * (v or 0.0) for v in potential.values), + unit=SeriesUnit.W, + ) + grid = days_series(date(2026, 6, 21), 1, lambda _b: 0.0) + assert not clipped_slots(pv, tz=BERLIN, grid=grid, export_limit=0.0, reference=potential) diff --git a/tests/forecast/test_pv_learning.py b/tests/forecast/test_pv_learning.py new file mode 100644 index 0000000..a7e278c --- /dev/null +++ b/tests/forecast/test_pv_learning.py @@ -0,0 +1,279 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The nightly PV learning job and the service's calibrated blend (WP5 part B).""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from datetime import UTC, date, datetime, time, timedelta +from pathlib import Path +from typing import Any +from zoneinfo import ZoneInfo + +import pytest + +from suncore.core.forecast import HorizonBucket +from suncore.core.measurements import SITE_GRID_W, SITE_PV_W +from suncore.core.series import QuantileSeries, SeriesUnit, TimeRange, TimeSeries +from suncore.forecast.accuracy import load_accuracy +from suncore.forecast.pv.adapter import FetchReport, FetchStatus +from suncore.forecast.pv.calibration import BiasTable, CellStats, load_bias +from suncore.forecast.pv.clearsky import ArrayGeometry, PlantGeometry, clear_sky_slot_w +from suncore.forecast.pv.learning import ( + BLENDED_SOURCE_ID, + LearningContext, + local_days, + mean_reference, + run_nightly, +) +from suncore.forecast.pv.service import BlendContext, PvForecastService, SourceSettings +from suncore.storage import Database, LearnedParamStore +from suncore.storage.timeseries import TimeseriesStore, open_timeseries + +BERLIN = ZoneInfo("Europe/Berlin") +Q = timedelta(minutes=15) +PLANT = PlantGeometry(48.0, 11.0, (ArrayGeometry(8.0, 180.0, 0.0),)) +FIRST = date(2026, 6, 18) +NOW = datetime.combine(date(2026, 6, 21), time(1, 0), tzinfo=BERLIN) + + +def midnight(day: date) -> datetime: + return datetime.combine(day, time(), tzinfo=BERLIN).astimezone(UTC) + + +def days_series(first: date, days: int, fn: Any, step: timedelta = Q) -> TimeSeries: + start = midnight(first).astimezone(UTC) + count = (midnight(first + timedelta(days=days)) - midnight(first)) // step + return TimeSeries( + start=start, + step=step, + values=tuple(fn(start + step * i) for i in range(count)), + unit=SeriesUnit.W, + ) + + +def sun(kt: float) -> Any: + return lambda begin: kt * clear_sky_slot_w(PLANT, begin, Q) + + +@pytest.fixture +async def store(tmp_path: Path) -> AsyncIterator[TimeseriesStore]: + db = await open_timeseries(tmp_path / "timeseries.db") + yield TimeseriesStore(db) + await db.close() + + +@pytest.fixture +async def learned(tmp_path: Path) -> AsyncIterator[LearnedParamStore]: + db = await Database.open(tmp_path / "suncore.db") + yield LearnedParamStore(db, clock=lambda: NOW) + await db.close() + + +def test_local_days_are_complete_and_dst_aware() -> None: + start = datetime(2026, 3, 28, 12, tzinfo=BERLIN) + days = local_days(start, datetime(2026, 3, 31, tzinfo=BERLIN), BERLIN) + assert [d.day for d in days] == [date(2026, 3, 29), date(2026, 3, 30)] + assert days[0].end - days[0].start == timedelta(hours=23) # spring forward + + +def test_mean_reference_averages_what_is_known() -> None: + window = days_series(FIRST, 1, lambda _b: 0.0) + a = TimeSeries(start=window.start, step=Q, values=(100.0, None, 300.0), unit=SeriesUnit.W) + b = TimeSeries(start=window.start + Q, step=Q, values=(200.0, 500.0), unit=SeriesUnit.W) + ref = mean_reference(window, [a, b]) + assert ref is not None + assert ref.values[:4] == (100.0, 200.0, 400.0, None) + assert mean_reference(window, []) is None + + +async def test_the_nightly_job_skips_clipping_scores_the_blend_and_learns_bias( + store: TimeseriesStore, learned: LearnedParamStore +) -> None: + limit = 3000.0 + potential = days_series(FIRST, 3, sun(0.8)) + pv = TimeSeries( + start=potential.start, + step=Q, + values=tuple(min(v or 0.0, limit) for v in potential.values), + unit=SeriesUnit.W, + ) + grid = TimeSeries( + start=pv.start, step=Q, values=tuple(-(v or 0.0) for v in pv.values), unit=SeriesUnit.W + ) + await store.store_quarter_hours(SITE_PV_W, pv) + await store.store_quarter_hours(SITE_GRID_W, grid) + # a source that is right (where nothing is clipped) issues every day at 23:00 for 48 h + for i in range(-1, 3): + issued = midnight(FIRST + timedelta(days=i)) - timedelta(hours=1) + series = days_series(FIRST + timedelta(days=i), 2, sun(0.8)) + await store.store_forecast("exact.pv", "pv", issued, QuantileSeries.deterministic(series)) + await store.store_forecast( + BLENDED_SOURCE_ID, "pv", issued, QuantileSeries.deterministic(series) + ) + + result = await run_nightly( + store, + learned, + ["exact.pv"], + NOW, + LearningContext(tz=BERLIN, plant=PLANT, export_limit_w=limit), + ) + assert result.clipped_slots > 0 + stored = await load_accuracy(learned) + assert set(stored) == {"exact.pv", BLENDED_SOURCE_ID} + for item in stored.values(): + assert item.records + for record in item.records: + assert abs(record.bias_w) < 1e-6 # the clipped quarter hours were left out + table = (await load_bias(learned))["exact.pv"] + assert table.days_with_data == 3 + assert table.learned_until == midnight(date(2026, 6, 21)) + assert table.mode == "clear_sky" + assert table.cells + for cell in table.cells.values(): + assert cell.actual_wh == pytest.approx(cell.forecast_wh) + + again = await run_nightly( + store, learned, ["exact.pv"], NOW, LearningContext(BERLIN, PLANT, limit) + ) + assert again.bias["exact.pv"].days_with_data == 3 # nothing is learnt twice + + +async def test_the_nightly_job_never_raises(store: TimeseriesStore) -> None: + class Broken: + async def put(self, *_a: object, **_k: object) -> None: + raise OSError + + async def in_namespace(self, _ns: str) -> list[Any]: + raise OSError + + result = await run_nightly(store, Broken(), ["a.pv"], NOW, LearningContext(BERLIN, None, None)) + assert result.accuracy == {} + + +# --- the service ------------------------------------------------------------------------ + + +class Source: + def __init__(self, source_id: str, series: QuantileSeries) -> None: + self.source_id = source_id + self.watched_entities: frozenset[str] = frozenset() + self.last_report = FetchReport(FetchStatus.OK) + self._series = series + + async def fetch(self, _window: TimeRange) -> QuantileSeries: + return self._series + + +class Store: + def __init__(self) -> None: + self.ids: list[str] = [] + + async def store_forecast(self, source_id: str, *_a: object) -> int: + self.ids.append(source_id) + return 1 + + +def biased_table(days: int) -> BiasTable: + cells = { + f"{h:02d}.jja.{w}": CellStats(n=1000, forecast_wh=1000, actual_wh=770) + for h in range(24) + for w in ("clear", "mixed", "overcast") + } + return BiasTable("high.pv", "clear_sky", cells, days_with_data=days) + + +async def test_the_service_blends_calibrates_and_stores_the_blend() -> None: + now = datetime(2026, 6, 21, 6, tzinfo=UTC) + day = days_series(date(2026, 6, 21), 2, sun(0.8)) + high = TimeSeries( + start=day.start, step=Q, values=tuple((v or 0) * 1.3 for v in day.values), unit=SeriesUnit.W + ) + store = Store() + service = PvForecastService( + [ + Source("high.pv", QuantileSeries.deterministic(high)), + Source("ok.pv", QuantileSeries.deterministic(day)), + ], + clock=lambda: now, + store=store, + settings={"ok.pv": SourceSettings("manual", 0.5)}, + min_history_days=30, + context=lambda: BlendContext(tz=BERLIN, plant=PLANT), + ) + await service.refresh() + assert BLENDED_SOURCE_ID in store.ids + + service.set_learning({}, {"high.pv": biased_table(29)}) + collecting = service.blended(now) + assert collecting is not None + assert not collecting.calibration_active + assert collecting.series == collecting.result.series + assert collecting.series.p50 == collecting.uncalibrated.p50 + assert collecting.calibrated is not None + weights = collecting.result.weights[0] + assert weights.horizon is HorizonBucket.H0_6 + assert weights.weights == {"ok.pv": pytest.approx(0.5), "high.pv": pytest.approx(0.5)} + + service.set_learning({}, {"high.pv": biased_table(30)}) + active = service.blended(now) + assert active is not None + assert active.calibration_active + assert active.series.p50 == active.calibrated.p50 if active.calibrated else False + noon = (datetime(2026, 6, 21, 11, tzinfo=UTC) - active.series.start) // Q + truth = day.values[(datetime(2026, 6, 21, 11, tzinfo=UTC) - day.start) // Q] + raw = active.uncalibrated.p50.values[noon] + fixed = active.series.p50.values[noon] + assert truth is not None + assert raw is not None + assert fixed is not None + assert abs(fixed - truth) < abs(raw - truth) / 5 + + view = service.view(48) + assert view.blended is not None + assert view.blended.calibration_active + assert [c.reason for c in view.blended.calibration] == ["active", "no_history"] + assert view.blended.uncalibrated_p50_w != view.blended.p50_w + + +async def test_a_stale_source_leaves_the_blend_while_a_fresh_one_exists() -> None: + clock = [datetime(2026, 6, 21, 4, tzinfo=UTC)] + day = days_series(date(2026, 6, 21), 2, sun(0.8)) + old_source = Source("old.pv", QuantileSeries.deterministic(day)) + new_source = Source("new.pv", QuantileSeries.deterministic(day)) + service = PvForecastService( + [old_source, new_source], + clock=lambda: clock[0], + context=lambda: BlendContext(tz=BERLIN, plant=PLANT), + ) + await service.refresh() + both = service.blended() + assert both is not None + assert both.result.sources == ("old.pv", "new.pv") + assert not both.stale + + # 13 h later only new.pv still delivers + clock[0] += timedelta(hours=13) + old_source.last_report = FetchReport(FetchStatus.NO_DATA) + old_source._series = QuantileSeries.deterministic( + TimeSeries(start=day.start, step=Q, values=(), unit=SeriesUnit.W) + ) + await service.refresh() + fresh = service.blended() + assert fresh is not None + assert fresh.result.sources == ("new.pv",) + assert fresh.excluded == ("old.pv",) + assert not fresh.stale + + # another 13 h without any delivery: both are stale, used anyway and flagged + clock[0] += timedelta(hours=13) + new_source._series = old_source._series + await service.refresh() + stale = service.blended() + assert stale is not None + assert stale.stale + assert stale.result.sources == ("old.pv", "new.pv") + view = service.view(24) + assert view.blended is not None + assert view.blended.stale + assert view.blended.excluded_sources == [] diff --git a/tests/forecast/test_pv_parsing.py b/tests/forecast/test_pv_parsing.py new file mode 100644 index 0000000..cdb05b6 --- /dev/null +++ b/tests/forecast/test_pv_parsing.py @@ -0,0 +1,292 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Parsing of the documented PV forecast shapes into W quantile series (Phase 4 WP5).""" + +from __future__ import annotations + +from datetime import UTC, date, datetime, timedelta + +import pytest + +from suncore.core.series import Grid, SeriesUnit, resample_power +from suncore.core.units import Unit +from suncore.forecast.pv.parsing import ( + RawPoint, + build_quantiles, + crop, + dig, + dig_response, + empty_series, + has_known, + infer_step, + list_points, + mapping_points, + number, + parse_timestamp, + regularize_energy, +) +from suncore.templates.models import SeriesFields +from tests.forecast.conftest import BERLIN, local_day_slots, solcast_entries + +HALF_HOUR = timedelta(minutes=30) +QUARTER = timedelta(minutes=15) +HOUR = timedelta(hours=1) +SOLCAST_FIELDS = SeriesFields( + timestamp_field="period_start", + value_field="pv_estimate", + p10_field="pv_estimate10", + p90_field="pv_estimate90", + value_unit="kW", +) + + +def solcast_series(*days: date, kw=None): # type: ignore[no-untyped-def] + entries = [e for day in days for e in solcast_entries(day, kw)] + parsed = list_points(entries, SOLCAST_FIELDS) + assert parsed.skipped == 0 + return build_quantiles(parsed.points, Unit.KW, HALF_HOUR) + + +def test_solcast_day_kw_becomes_w_on_the_native_grid() -> None: + series = solcast_series(date(2026, 6, 21)) + assert series is not None + assert series.unit is SeriesUnit.W + assert series.step == HALF_HOUR + assert len(series) == 48 + assert series.start == datetime(2026, 6, 20, 22, 0, tzinfo=UTC) # local midnight, UTC+2 + noon = series.p50.values[2 * 12 + 4] # local 12:00 -> 2.0 kW + assert noon == 2000.0 + assert series.p10.values[2 * 12 + 4] == pytest.approx(1200.0) + assert series.p90.values[2 * 12 + 4] == pytest.approx(2400.0) + assert series.p50.values[0] == 0.0 # night is a known zero, not a gap + + +def test_solcast_energy_is_the_sum_of_kw_times_half_an_hour() -> None: + series = solcast_series(date(2026, 6, 21)) + assert series is not None + kw_sum = 20 * 2.0 # 10 h at 2 kW = 20 slots of 2 kW + assert series.p50.total() == pytest.approx(kw_sum * 0.5 * 1000) # Wh + + +def test_spring_forward_day_has_46_slots_and_no_gap() -> None: + day = date(2026, 3, 29) # Europe/Berlin: 02:00 -> 03:00 + assert len(local_day_slots(day, 30)) == 46 + series = solcast_series(day, kw=lambda _: 1.0) + assert series is not None + assert len(series) == 46 + assert all(v == 1000.0 for v in series.p50.values) # no None: the missing hour does not exist + + +def test_fall_back_day_has_50_slots_with_distinct_instants() -> None: + day = date(2026, 10, 25) # Europe/Berlin: 03:00 -> 02:00 + entries = solcast_entries(day, kw=lambda _: 1.0) + assert len(entries) == 50 + offsets = {e["period_start"][-6:] for e in entries} + assert offsets == {"+02:00", "+01:00"} # the repeated hour carries both offsets + series = solcast_series(day, kw=lambda _: 1.0) + assert series is not None + assert len(series) == 50 + assert all(v == 1000.0 for v in series.p50.values) + + +def test_day_sensors_join_into_one_series_and_a_missing_day_is_a_gap() -> None: + # today, tomorrow and d4 (d3's sensor is unavailable): 48 slots of None in between + series = solcast_series(date(2026, 6, 21), date(2026, 6, 22), date(2026, 6, 24)) + assert series is not None + assert len(series) == 4 * 48 + assert series.p50.values[48 + 24] == 2000.0 # tomorrow noon + gap = series.p50.values[2 * 48 : 3 * 48] + assert gap == (None,) * 48 # never a copy of the neighbour's value + assert series.p50.values[3 * 48 + 24] == 2000.0 + + +def test_dst_day_joins_without_a_gap_to_the_next_day() -> None: + series = solcast_series( + date(2026, 3, 28), date(2026, 3, 29), date(2026, 3, 30), kw=lambda _: 1.0 + ) + assert series is not None + assert len(series) == 48 + 46 + 48 + assert None not in series.p50.values + + +def test_irregular_timestamps_are_refused_not_guessed() -> None: + entries = solcast_entries(date(2026, 6, 21)) + entries[5]["period_start"] = "2026-06-21T02:40:00+02:00" # off the 30-minute grid + parsed = list_points(entries, SOLCAST_FIELDS) + with pytest.raises(ValueError, match="neither"): + build_quantiles(parsed.points, Unit.KW, HALF_HOUR) + + +def test_partial_entries_keep_their_slot_and_unknown_quantiles() -> None: + entries = solcast_entries(date(2026, 6, 21)) + del entries[24]["pv_estimate10"] # no P10 for this slot + entries[25]["pv_estimate"] = "unavailable" # unparsable value + entries[26]["pv_estimate"] = -3.0 # a negative forecast is invalid data + entries[27] = "garbage" # not an object + entries[28]["period_start"] = "2026-06-21 12:00" # no offset: refused + parsed = list_points(entries, SOLCAST_FIELDS) + assert parsed.skipped == 2 + series = build_quantiles(parsed.points, Unit.KW, HALF_HOUR) + assert series is not None + assert series.p50.values[24] == 2000.0 + assert series.p10.values[24] is None + assert series.p50.values[25] is None + assert series.p10.values[25] is None + assert series.p50.values[26] is None + assert series.p50.values[27] is None + assert series.p50.values[28] is None + assert series.p50.values[29] == 2000.0 + + +def test_inverted_quantiles_are_clamped_to_p50() -> None: + points = [RawPoint(datetime(2026, 6, 21, 10, tzinfo=UTC), 2.0, 2.5, 1.5)] + series = build_quantiles(points, Unit.KW, HALF_HOUR) + assert series is not None + assert (series.p10.values[0], series.p50.values[0], series.p90.values[0]) == (2000, 2000, 2000) + + +def test_a_source_without_quantile_fields_has_equal_quantiles() -> None: + points = [RawPoint(datetime(2026, 6, 21, 10, tzinfo=UTC), 400.0)] + series = build_quantiles(points, Unit.WH, QUARTER) + assert series is not None + assert series.p10 == series.p50 == series.p90 + + +def test_energy_per_slot_converts_to_power_and_preserves_the_energy() -> None: + start = datetime(2026, 6, 21, 10, tzinfo=UTC) + wh = [0.0, 125.0, 250.0, 500.0] # Wh per 15 minutes (Open-Meteo wh_period_15m) + points = [RawPoint(start + QUARTER * i, v) for i, v in enumerate(wh)] + series = build_quantiles(points, Unit.WH, QUARTER) + assert series is not None + assert series.p50.values == (0.0, 500.0, 1000.0, 2000.0) # W = Wh * 4 + assert series.p50.total() == pytest.approx(sum(wh)) + hourly = [RawPoint(start + HOUR * i, v) for i, v in enumerate([1500.0, 3000.0])] # Wh per hour + h = build_quantiles(hourly, Unit.WH, HOUR) + assert h is not None + assert h.p50.values == (1500.0, 3000.0) + k = build_quantiles([RawPoint(start, 1.5)], Unit.KWH, HOUR) # 1.5 kWh in an hour + assert k is not None + assert k.p50.values == (1500.0,) + + +def test_refining_the_hourly_forecast_onto_the_planning_grid_keeps_the_energy() -> None: + start = datetime(2026, 6, 21, 10, tzinfo=UTC) + points = [RawPoint(start + HOUR * i, v) for i, v in enumerate([1000.0, 2000.0, 500.0])] + series = build_quantiles(points, Unit.WH, HOUR) + assert series is not None + fine = resample_power(series.p50, Grid(start=start, step=QUARTER, count=12)) + assert fine.total() == pytest.approx(series.p50.total()) + assert fine.values[:4] == (1000.0,) * 4 + + +def test_missing_is_zero_fills_between_known_periods_only() -> None: + start = datetime(2026, 6, 21, 4, tzinfo=UTC) + # two missing periods between known ones are read as 0 (the general rule; Forecast.Solar + # itself only ever drops the zero at 00:00:00) + points = [RawPoint(start, 100.0), RawPoint(start + 3 * HOUR, 300.0)] + filled = build_quantiles(points, Unit.WH, HOUR, missing_is_zero=True) + assert filled is not None + assert filled.p50.values == (100.0, 0.0, 0.0, 300.0) + plain = build_quantiles(points, Unit.WH, HOUR) + assert plain is not None + assert plain.p50.values == (100.0, None, None, 300.0) + assert has_known(filled) + assert not has_known(empty_series(start, HOUR)) + + +def test_infer_step_from_spacing_never_assumes() -> None: + start = datetime(2026, 6, 21, 4, tzinfo=UTC) + hourly = [RawPoint(start + HOUR * i, 1.0) for i in (0, 1, 3, 4)] # one missing hour + assert infer_step(hourly, QUARTER) == HOUR + half = [RawPoint(start + HALF_HOUR * i, 1.0) for i in range(3)] + assert infer_step(half, HOUR) == HALF_HOUR + assert infer_step([RawPoint(start, 1.0)], HOUR) == HOUR # one point: the default + + +def test_mapping_points_and_timestamp_rules() -> None: + parsed = mapping_points( + { + "2026-06-21T10:00:00+02:00": 120.5, + "2026-06-21T10:15:00+02:00": "130", + "2026-06-21T10:30:00": 5, # naive + "nonsense": 1, + "2026-06-21T10:45:00+02:00": None, + } + ) + assert parsed.skipped == 2 + assert [p.value for p in parsed.points] == [120.5, 130.0, None] + assert mapping_points(None).points == () + assert parse_timestamp(datetime(2026, 1, 1)) is None # noqa: DTZ001 + assert number(True) is None + assert number(float("nan")) is None + assert number("x") is None + + +def test_dig_and_response_paths() -> None: + assert dig({"a": {"b": [1]}}, "a.b") == [1] + assert dig({"a": 1}, "a.b") is None + assert dig("x", None) == "x" + response = {"prices": {"Home": [{"s": 1}], "Cabin": [{"s": 2}]}, "data": [{"s": 3}]} + assert dig_response(response, "prices.*") == [{"s": 1}, {"s": 2}] + assert dig_response(response, "data") == [{"s": 3}] + assert dig_response(response, "nothing") == [] + + +def test_crop_selects_overlapping_slots_and_never_pads() -> None: + series = solcast_series(date(2026, 6, 21)) + assert series is not None + start = series.start + timedelta(hours=10, minutes=10) # inside slot 20 + cut = crop(series, start, start + timedelta(hours=1)) + assert cut.start == series.start + timedelta(hours=10) + assert len(cut) == 3 # 10:10-11:10 overlaps the slots 10:00, 10:30 and 11:00 + before = crop(series, series.start - timedelta(days=1), series.start + HALF_HOUR) + assert len(before) == 1 + assert before.start == series.start + outside = crop(series, series.p50.end + HOUR, series.p50.end + 2 * HOUR) + assert len(outside) == 0 + + +def test_local_timestamps_keep_their_offsets_in_utc() -> None: + slots = local_day_slots(date(2026, 6, 21), 30) + assert slots[0].utcoffset() == timedelta(hours=2) + assert slots[0].tzinfo == BERLIN + + +# --- off-grid points of an energy mapping (energy/solar_forecast) ------------------------- + + +def test_an_off_grid_sunrise_key_never_makes_a_one_minute_grid() -> None: + start = datetime(2026, 6, 21, 4, tzinfo=UTC) + points = [RawPoint(start + HOUR * i, 100.0 * i) for i in range(6)] + sunrise = RawPoint(datetime(2026, 6, 21, 3, 41, 13, tzinfo=UTC), 0.0) + sunset = RawPoint(datetime(2026, 6, 21, 8, 12, 5, tzinfo=UTC), 40.0) + assert infer_step([*points, sunrise, sunset], HOUR) < HOUR # the old behaviour + regular = regularize_energy([*points, sunrise, sunset], HOUR) + assert regular.step == HOUR + assert regular.dropped == 1 + assert regular.merged == 1 + series = build_quantiles(regular.points, Unit.WH, regular.step) + assert series is not None + assert series.p50.values == (0.0, 100.0, 200.0, 300.0, 440.0, 500.0) + total_in = sum(p.value or 0.0 for p in [*points, sunrise, sunset]) + assert series.p50.total() == pytest.approx(total_in) + + +def test_an_off_grid_point_outside_the_series_opens_its_period() -> None: + start = datetime(2026, 6, 21, 4, tzinfo=UTC) + points = [RawPoint(start + HOUR * i, 10.0) for i in range(3)] + late = RawPoint(datetime(2026, 6, 21, 7, 30, tzinfo=UTC), 5.0) + regular = regularize_energy([*points, late], HOUR) + assert regular.step == HOUR # 07:30 lies on the 15-min grid but the hours dominate + assert [p.start.hour for p in regular.points] == [4, 5, 6, 7] + assert regular.points[-1].value == 5.0 + + +@pytest.mark.parametrize("minutes", [15, 30, 60]) +def test_regular_mappings_keep_their_period(minutes: int) -> None: + step = timedelta(minutes=minutes) + start = datetime(2026, 6, 21, 4, tzinfo=UTC) + points = [RawPoint(start + step * i, 1.0) for i in range(8)] + regular = regularize_energy(points, HOUR) + assert regular.step == step + assert (regular.dropped, regular.merged) == (0, 0) + assert regular.points == tuple(points) diff --git a/tests/forecast/test_pv_service.py b/tests/forecast/test_pv_service.py new file mode 100644 index 0000000..2456c14 --- /dev/null +++ b/tests/forecast/test_pv_service.py @@ -0,0 +1,423 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The PV forecast service: keep the last good data, gap filling, storage cadence (WP5).""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from suncore.core.series import QuantileSeries, SeriesUnit, TimeRange, TimeSeries +from suncore.forecast.pv.adapter import FetchReport, FetchStatus +from suncore.forecast.pv.service import ( + PvForecastService, + fill_from_previous, + materially_changed, +) +from suncore.storage import Database +from suncore.storage.timeseries import TimeseriesStore, open_timeseries + +T0 = datetime(2026, 6, 21, 6, 0, tzinfo=UTC) +HALF = timedelta(minutes=30) + + +def series( + values: list[float | None], start: datetime = T0, step: timedelta = HALF +) -> QuantileSeries: + ts = TimeSeries(start=start, step=step, values=tuple(values), unit=SeriesUnit.W) + return QuantileSeries.deterministic(ts) + + +def day(level: float, start: datetime = T0, slots: int = 96) -> QuantileSeries: + return series([level] * slots, start) + + +class Clock: + def __init__(self, now: datetime = T0) -> None: + self.now = now + + def __call__(self) -> datetime: + return self.now + + def advance(self, **kw: float) -> None: + self.now += timedelta(**kw) + + +class ScriptedSource: + """A source that returns what the test scripts (a series, or an exception).""" + + def __init__( + self, source_id: str = "solcast.pv", entities: frozenset[str] = frozenset({"sensor.a"}) + ) -> None: + self._id = source_id + self.watched_entities = entities + self.next: QuantileSeries | Exception = series([]) + self.last_report = FetchReport(FetchStatus.OK) + self.windows: list[TimeRange] = [] + + @property + def source_id(self) -> str: + return self._id + + async def fetch(self, window: TimeRange) -> QuantileSeries: + self.windows.append(window) + if isinstance(self.next, Exception): + raise self.next + return self.next + + +class RecordingStore: + """Records the stored issues; those of the blend are kept apart (``blended_calls``).""" + + def __init__(self) -> None: + self.calls: list[tuple[str, str, datetime, QuantileSeries]] = [] + self.blended_calls: list[tuple[str, str, datetime, QuantileSeries]] = [] + self.fail = False + + async def store_forecast(self, source_id, kind, issued_at, forecast): # type: ignore[no-untyped-def] + if self.fail: + msg = "disk full" + raise OSError(msg) + target = self.blended_calls if source_id == "blended" else self.calls + target.append((source_id, kind, issued_at, forecast)) + return len(forecast) + + +def make( + source: ScriptedSource, clock: Clock, store: RecordingStore | None = None, **kw: object +) -> PvForecastService: + return PvForecastService( + [source], + clock=clock, + store=store, + sleep=kw.pop("sleep", asyncio.sleep), # type: ignore[arg-type] + **kw, # type: ignore[arg-type] + ) + + +# --- keep the last good data ----------------------------------------------------------- + + +async def test_first_good_fetch_is_exposed_with_its_freshness() -> None: + clock, source = Clock(), ScriptedSource() + service = make(source, clock) + source.next = day(1000.0) + await service.refresh() + snap = service.snapshot("solcast.pv") + assert snap is not None + assert snap.series is not None + assert snap.fetched_at == T0 + assert snap.status == "ok" + view = service.view(24) + assert view.sources[0].stale is False + assert view.sources[0].age_s == 0 + assert view.sources[0].series is not None + assert view.sources[0].series.p50_w[0] == 1000.0 + assert source.windows[0].start == T0 + assert source.windows[0].end == T0 + timedelta(hours=48) + + +@pytest.mark.parametrize("bad", ["empty", "all_none", "past_only", "error"]) +async def test_a_bad_fetch_never_replaces_good_data(bad: str) -> None: + clock, source = Clock(), ScriptedSource() + service = make(source, clock) + source.next = day(1000.0) + await service.refresh() + good = service.snapshot("solcast.pv") + assert good is not None + kept = good.series + clock.advance(minutes=30) + if bad == "empty": + source.next = series([]) + source.last_report = FetchReport(FetchStatus.NO_DATA) + elif bad == "all_none": + source.next = series([None] * 10, clock.now) + source.last_report = FetchReport(FetchStatus.NO_DATA) + elif bad == "past_only": # the integration did not roll over: only slots that already ended + source.next = series([1.0] * 4, T0 - timedelta(hours=10)) + else: + source.next = RuntimeError("boom") + await service.refresh() + snap = service.snapshot("solcast.pv") + assert snap is not None + assert snap.series is kept # untouched + assert snap.fetched_at == T0 # the age keeps growing + assert snap.status in {"no_data", "error"} + assert service.view(24).sources[0].series is not None + + +async def test_stale_flag_after_three_missed_refreshes() -> None: + clock, source = Clock(), ScriptedSource() + service = make(source, clock, refresh_interval_s=1800.0) + assert service.view().sources[0].stale is True # never fetched + source.next = day(1000.0) + await service.refresh() + clock.advance(minutes=89) + assert service.view().sources[0].stale is False + clock.advance(minutes=2) + assert service.view().sources[0].stale is True + + +async def test_gap_filling_from_the_last_good_fetch_is_bounded() -> None: + clock, source = Clock(), ScriptedSource() + service = make(source, clock, gap_fill_max_age=timedelta(hours=2)) + source.next = day(1000.0) # 48 h + await service.refresh() + clock.advance(minutes=30) + # tomorrow's sensor flaps: the new fetch only knows the next 6 hours + source.next = series([2000.0] * 12, clock.now) + await service.refresh() + snap = service.snapshot("solcast.pv") + assert snap is not None + assert snap.series is not None + assert snap.filled_slots > 0 + assert snap.series.p50.values[0] == 2000.0 + assert snap.series.p50.values[12] == 1000.0 + assert len(snap.series) == 95 # extended to where the previous good fetch reached + # an hour later: past the bound after the last complete fetch -> no more filling + clock.advance(hours=2, minutes=1) + source.next = series([3000.0] * 12, T0 + timedelta(hours=3)) + await service.refresh() + again = service.snapshot("solcast.pv") + assert again is not None + assert again.series is not None + assert again.filled_slots == 0 + assert len(again.series) == 12 + + +def test_fill_from_previous_only_fills_unknown_future_slots_on_the_same_grid() -> None: + previous = series([1.0, 2.0, 3.0, 4.0]) + new = series([None, 9.0, None], T0 + HALF) # slots 1..3 of previous + now = T0 + HALF + timedelta(minutes=10) + filled, count = fill_from_previous(new, previous, now) + assert count == 2 + # slot 0 of new (T0+30 min .. 1 h) is current: filled from previous slot 1 = 2.0 + assert filled.p50.values == (2.0, 9.0, 4.0) + other_grid = series([1.0, 2.0], T0, timedelta(hours=1)) + assert fill_from_previous(new, other_grid, now) == (new, 0) + ended = fill_from_previous(series([None], T0), series([5.0]), T0 + timedelta(hours=1)) + assert ended[1] == 0 # a slot that already ended is not worth filling + + +# --- storage cadence ------------------------------------------------------------------ + + +async def test_storage_is_at_most_hourly_unless_the_forecast_changed_materially() -> None: + clock, source, store = Clock(), ScriptedSource(), RecordingStore() + service = make(source, clock, store=store) + source.next = day(1000.0) + await service.refresh() + assert len(store.calls) == 1 + name, kind, issued, issue = store.calls[0] + assert (name, kind, issued) == ("solcast.pv", "pv", T0) + assert issue.start == T0 + assert issue.p50.end <= T0 + timedelta(hours=48) + + for _ in range(5): # unchanged refreshes every 10 minutes + clock.advance(minutes=10) + await service.refresh() + assert len(store.calls) == 1 # 50 minutes, nothing stored + + clock.advance(minutes=10) # one hour since the first store + await service.refresh() + assert len(store.calls) == 2 + + clock.advance(minutes=5) + source.next = day( + 400.0, T0 + timedelta(hours=1) + ) # a big change, but only 5 minutes after the last store + await service.refresh() + assert len(store.calls) == 2 # the minimum gap holds + clock.advance(minutes=10) + await service.refresh() + assert len(store.calls) == 3 # changed materially and past the gap + assert store.calls[-1][3].p50.values[0] == 400.0 + + clock.advance(minutes=20) + source.next = day(410.0, T0 + timedelta(hours=1)) # 2.5 %: not material + await service.refresh() + assert len(store.calls) == 3 + + +async def test_a_day_of_five_minute_refreshes_stores_at_most_an_hourly_series() -> None: + clock, source, store = Clock(), ScriptedSource(), RecordingStore() + service = make(source, clock, store=store) + source.next = day(1000.0, slots=200) + for _ in range(24 * 12): + await service.refresh() + clock.advance(minutes=5) + assert len(store.calls) <= 25 + assert sum(len(call[3]) for call in store.calls) <= 25 * 96 + + +async def test_storage_failure_is_retried_and_never_raised() -> None: + clock, source, store = Clock(), ScriptedSource(), RecordingStore() + store.fail = True + service = make(source, clock, store=store) + source.next = day(1000.0) + await service.refresh() # does not raise + assert store.calls == [] + store.fail = False + clock.advance(minutes=1) + await service.refresh() + assert len(store.calls) == 1 # last_stored_at was not set, so it is retried + + +async def test_what_is_stored_is_what_the_source_said_not_the_filled_series() -> None: + clock, source, store = Clock(), ScriptedSource(), RecordingStore() + service = make(source, clock, store=store, store_interval_s=60.0, min_store_gap_s=0.0) + source.next = day(1000.0) + await service.refresh() + clock.advance(minutes=2) + source.next = series([2000.0] * 4, T0 + HALF) + await service.refresh() + assert len(store.calls[-1][3]) == 4 # not extended by the previous fetch's tail + + +def test_materially_changed_ignores_unknown_slots_and_night_noise() -> None: + a, b = day(1000.0), day(1100.0) + assert not materially_changed(a, b, T0) + assert materially_changed(a, day(300.0), T0) + assert not materially_changed(day(0.0), day(1.0), T0) # tiny absolute change at night + assert not materially_changed(series([None] * 10), day(500.0), T0) # nothing comparable + + +# --- real store round trip ------------------------------------------------------------- + + +@pytest.fixture +async def tsdb(tmp_path: Path) -> AsyncIterator[Database]: + db = await open_timeseries(tmp_path / "timeseries.db") + yield db + await db.close() + + +async def test_the_issue_is_readable_from_timeseries_db(tsdb: Database) -> None: + clock, source = Clock(), ScriptedSource() + store = TimeseriesStore(tsdb) + service = make(source, clock, store=store) + source.next = day(1234.0) + await service.refresh() + issues = await store.forecast_issues("solcast.pv", "pv") + assert issues == [T0] + back = await store.read_forecast("solcast.pv", "pv", T0) + assert back is not None + assert back.step == HALF + assert back.p50.values[0] == 1234.0 + + +# --- loop and wake ------------------------------------------------------------------ + + +async def test_wake_refreshes_a_source_without_entities() -> None: + clock, source = Clock(), ScriptedSource(entities=frozenset()) + source.next = day(1000.0) + gate = asyncio.Event() + + async def sleep(delay: float) -> None: + if delay > 100: + await gate.wait() + + service = make(source, clock, sleep=sleep, refresh_interval_s=1800.0) + task = asyncio.create_task(service.run()) + try: + for _ in range(20): + await asyncio.sleep(0) + assert len(source.windows) == 1 + service.on_entities(["sensor.a"]) # an energy/solar_forecast source watches nothing + for _ in range(20): + await asyncio.sleep(0) + assert len(source.windows) == 1 + service.wake() + for _ in range(30): + await asyncio.sleep(0) + assert len(source.windows) == 2 + finally: + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + +async def test_entity_changes_wake_the_loop_and_unrelated_ones_do_not() -> None: + clock, source = Clock(), ScriptedSource() + source.next = day(1000.0) + sleeps: list[float] = [] + gate = asyncio.Event() + + async def sleep(delay: float) -> None: + sleeps.append(delay) + if delay > 100: + await gate.wait() # the refresh interval: never ends by itself + + service = make(source, clock, sleep=sleep, refresh_interval_s=1800.0) + task = asyncio.create_task(service.run()) + try: + for _ in range(20): + await asyncio.sleep(0) + assert len(source.windows) == 1 + service.on_entities(["sensor.unrelated"]) + for _ in range(20): + await asyncio.sleep(0) + assert len(source.windows) == 1 + service.on_entities(["sensor.a"]) + for _ in range(30): + await asyncio.sleep(0) + assert len(source.windows) == 2 # woken, debounced by a short sleep + assert 30.0 in sleeps + finally: + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + +def test_no_blend_without_data_and_the_watched_entities_of_all_sources() -> None: + clock = Clock() + first, second = ScriptedSource("a.pv"), ScriptedSource("b.pv", frozenset({"sensor.b"})) + service = PvForecastService([first, second], clock=clock) + assert service.blended() is None + assert service.watched_entities == {"sensor.a", "sensor.b"} + + +async def test_stopping_the_service_wins_over_a_simultaneous_wake() -> None: + clock, source = Clock(), ScriptedSource() + source.next = day(1000.0) + gate = asyncio.Event() + + async def sleep(delay: float) -> None: + if delay > 100: + await gate.wait() + + service = make(source, clock, sleep=sleep) + task = asyncio.create_task(service.run()) + for _ in range(20): + await asyncio.sleep(0) + service.on_entities(["sensor.a"]) # a wake ... + task.cancel() # ... and a stop in the same loop iteration + async with asyncio.timeout(2): + with pytest.raises(asyncio.CancelledError): + await task + + +async def test_a_change_during_the_refresh_is_not_lost() -> None: + clock, source = Clock(), ScriptedSource() + source.next = day(1000.0) + sleeps: list[float] = [] + + async def sleep(delay: float) -> None: + sleeps.append(delay) + if delay > 100: + await asyncio.Event().wait() + + service = make(source, clock, sleep=sleep) + service.on_entities(["sensor.a"]) # arrives before the loop reaches its sleep + task = asyncio.create_task(service.run()) + for _ in range(30): + await asyncio.sleep(0) + assert len(source.windows) >= 2 + assert 1800.0 not in sleeps[:1] + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task diff --git a/tests/ha/fakes.py b/tests/ha/fakes.py index 114e758..8d681d1 100644 --- a/tests/ha/fakes.py +++ b/tests/ha/fakes.py @@ -112,6 +112,8 @@ def __init__(self, token: str = TOKEN) -> None: self.device_registry: list[dict[str, Any]] = [] self.area_registry: list[dict[str, Any]] = [] self.energy_prefs: dict[str, Any] | None = None # None = "No prefs" (not_found) + # energy/solar_forecast: config entry id -> {wh_hours: {iso: Wh}} (research: types.py). + self.solar_forecast: Any = {} # recorder/statistics_during_period: statistic_id -> rows (ms epoch, as HA answers). self.statistics: dict[str, list[dict[str, Any]]] = {} self.service_response: Callable[[dict[str, Any]], Any] = lambda msg: {"echo": msg} @@ -275,6 +277,8 @@ def ok(result: Any = None) -> dict[str, Any]: ) else: await self._reply(ws, ok(self.energy_prefs)) + elif kind == "energy/solar_forecast": + await self._reply(ws, ok(self.solar_forecast)) elif kind == "recorder/statistics_during_period": wanted_ids = msg["statistic_ids"] result = {i: self.statistics[i] for i in wanted_ids if self.statistics.get(i)} diff --git a/tests/ha/test_solar_forecast.py b/tests/ha/test_solar_forecast.py new file mode 100644 index 0000000..9a0a7b2 --- /dev/null +++ b/tests/ha/test_solar_forecast.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +"""``energy/solar_forecast``: the read-only command, its parsing and the fake (Phase 4 WP5). + +Shape (``docs/research/ha-websocket-api.md`` "energy/solar_forecast"): +``{"": {"wh_hours": {"": }}}``, ``{}`` when no +forecast integration is linked in the Energy dashboard. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator +from datetime import UTC, datetime + +import pytest + +from suncore.ha import fetch_solar_forecast +from suncore.ha.connection import CommandError, ProtocolError +from suncore.ha.solar_forecast import parse_solar_forecast +from suncore.ha.transport import DirectTransport +from suncore.ha.websocket import WebSocketConnection +from suncore.sim.fake_ha import FakeHAConnection +from suncore.sim.scenario import load_scenario +from tests.ha.fakes import TOKEN, FakeClock, FakeHA + +HOURS = {"2026-06-21T10:00:00+02:00": 1200, "2026-06-21T11:00:00+02:00": 1500.5} + + +@pytest.fixture +async def ha() -> AsyncIterator[FakeHA]: + server = FakeHA() + await server.start() + yield server + await server.stop() + + +@pytest.fixture +async def conn(ha: FakeHA) -> AsyncIterator[WebSocketConnection]: + clock = FakeClock() + connection = WebSocketConnection( + DirectTransport.from_base_url(ha.base_url, TOKEN), + clock=clock.monotonic, + sleep=clock.sleep, + rng=lambda: 0.5, + heartbeat_interval=777.0, + heartbeat_timeout=666.0, + resync_interval=555.0, + command_timeout=888.0, + connect_timeout=999.0, + ) + await asyncio.wait_for(connection.connect(), 5) + yield connection + await connection.close() + + +async def test_the_command_is_sent_without_fields_and_parsed( + conn: WebSocketConnection, ha: FakeHA +) -> None: + ha.solar_forecast = {"01JABC": {"wh_hours": HOURS}, "01JDEF": {"wh_hours": {}}} + got = await fetch_solar_forecast(conn) + assert got == {"01JABC": {k: float(v) for k, v in HOURS.items()}, "01JDEF": {}} + sent = next(m for m in ha.received if m["type"] == "energy/solar_forecast") + assert set(sent) == {"id", "type"} # no fields, no admin requirement + + +async def test_nothing_linked_is_an_empty_result(conn: WebSocketConnection, ha: FakeHA) -> None: + ha.solar_forecast = {} + assert await conn.energy_solar_forecast() == {} + + +async def test_wrong_shapes_and_errors(conn: WebSocketConnection, ha: FakeHA) -> None: + ha.solar_forecast = ["not", "an", "object"] + with pytest.raises(ProtocolError): + await conn.energy_solar_forecast() + ha.fail_commands["energy/solar_forecast"] = ("unknown_command", "Unknown command.") + with pytest.raises(CommandError) as err: + await conn.energy_solar_forecast() + assert err.value.code == "unknown_command" + + +def test_parse_drops_malformed_entries_and_values() -> None: + parsed = parse_solar_forecast( + { + "a": { + "wh_hours": { + "2026-06-21T10:00:00+02:00": 5, + "bad": "x", + "k": True, + "n": float("nan"), + } + }, + "b": {"nope": {}}, + "c": "junk", + } + ) + assert parsed == {"a": {"2026-06-21T10:00:00+02:00": 5.0}} + + +async def test_fake_serves_the_two_linked_entries_and_respects_the_dashboard() -> None: + never = asyncio.Event() + + async def sleep(_: float) -> None: + await never.wait() + + def build(*, configured: bool) -> FakeHAConnection: + return FakeHAConnection( + load_scenario("summer_day"), + clock=lambda: datetime(2026, 6, 21, tzinfo=UTC), + sleep=sleep, + start_offset_s=13 * 3600, + energy_prefs_configured=configured, + ) + + conn = build(configured=True) + await conn.connect() + entries = await conn.energy_solar_forecast() + prefs = await conn.energy_get_prefs() + assert prefs is not None + linked = next(s for s in prefs["energy_sources"] if s["type"] == "solar") + assert set(entries) == set(linked["config_entry_solar_forecast"]) + hourly = entries["sim_entry_forecast_solar"] + # forecast_solar/energy.py drops only a 0 Wh entry at exactly 00:00:00 + assert not any(v == 0 and k.endswith("T00:00:00+02:00") for k, v in hourly.items()) + assert any(v == 0 for v in hourly.values()) # the other night hours are listed + half_hourly = entries["sim_entry_solcast_solar"] + assert len(half_hourly) == 96 + await conn.close() + off = build(configured=False) + await off.connect() + assert await off.energy_solar_forecast() == {} + await off.close() diff --git a/tests/safety/test_boundary.py b/tests/safety/test_boundary.py index 28daa62..5685e7e 100644 --- a/tests/safety/test_boundary.py +++ b/tests/safety/test_boundary.py @@ -33,9 +33,11 @@ only in ``suncore.ha`` (implementation, rate limited per action), ``suncore.sim`` (fake) and the one adapter wiring module ``suncore.app.adapters`` (Phase 4); no other part of ``suncore.app`` may use it. ``READ_ONLY_ACTIONS`` is the fixed allowlist. - -8. ``statistics_during_period`` (read-only recorder statistics) is used only in ``suncore.ha``, +8. ``statistics_during_period`` (read-only recorder statistics) is used only in ``suncore.ha`` and ``suncore.sim`` (``suncore.app.history`` goes through ``suncore.ha.statistics``). +9. ``energy_solar_forecast`` (the read-only ``energy/solar_forecast`` command) is used only in + ``suncore.ha`` and ``suncore.sim`` (``suncore.app.adapters`` goes through + ``suncore.ha.fetch_solar_forecast``). It is a read: no write path changes (Phase 4 WP5). The scan is robust to packages that are still empty (``suncore/ha`` until WP3) because it only looks at the files that exist. @@ -68,6 +70,7 @@ WRITE_CAPABILITY_NAMES = frozenset({"ServiceCaller", "ServiceCallerProvider"}) SAFETY_FORBIDDEN_IMPORTS = ("suncore.core.config", "suncore.storage") STATISTICS_OWNERS = ("suncore.ha", "suncore.sim") +SOLAR_FORECAST_OWNERS = ("suncore.ha", "suncore.sim") ACTION_RESPONSE_OWNERS = ("suncore.ha", "suncore.sim", "suncore.app.adapters") @@ -350,6 +353,16 @@ def test_statistics_during_period_is_confined(path: Path) -> None: assert not offenders, f"statistics_during_period referenced outside ha/sim: {offenders}" +@pytest.mark.parametrize("path", SOURCES, ids=lambda p: str(p.relative_to(ROOT))) +def test_energy_solar_forecast_is_confined(path: Path) -> None: + """Rule 9: the Energy dashboard's forecast read lives in ha (+ the fake) only.""" + module = module_name(path) + if any(in_package(module, owner) for owner in SOLAR_FORECAST_OWNERS): + return + offenders = fmt(path, attribute_refs(parse(path), "energy_solar_forecast")) + assert not offenders, f"energy_solar_forecast referenced outside ha/sim: {offenders}" + + def test_action_response_owner_is_the_adapter_module_not_the_whole_app() -> None: """Rule 7 admits ``suncore.app.adapters`` only: its siblings in ``suncore.app`` stay out.""" assert "suncore.app" not in ACTION_RESPONSE_OWNERS diff --git a/tests/scenarios/forecast/forecast_solar.pv.yaml b/tests/scenarios/forecast/forecast_solar.pv.yaml new file mode 100644 index 0000000..fa1b407 --- /dev/null +++ b/tests/scenarios/forecast/forecast_solar.pv.yaml @@ -0,0 +1,4 @@ +# No entity roles: the series comes from `energy/solar_forecast` (config entry chosen by the user). +template: forecast_solar.pv +sim_scenario: summer_day +roles: {} diff --git a/tests/scenarios/forecast/generic.pv_forecast.yaml b/tests/scenarios/forecast/generic.pv_forecast.yaml new file mode 100644 index 0000000..3a2d299 --- /dev/null +++ b/tests/scenarios/forecast/generic.pv_forecast.yaml @@ -0,0 +1,7 @@ +template: generic.pv_forecast +sim_scenario: summer_day +roles: + forecast: + entity: sensor.sim_pv_forecast_template + unit: kWh + criteria: [domain, platforms] diff --git a/tests/scenarios/forecast/open_meteo_solar_forecast.pv.yaml b/tests/scenarios/forecast/open_meteo_solar_forecast.pv.yaml new file mode 100644 index 0000000..7a0ebda --- /dev/null +++ b/tests/scenarios/forecast/open_meteo_solar_forecast.pv.yaml @@ -0,0 +1,15 @@ +template: open_meteo_solar_forecast.pv +sim_scenario: summer_day +roles: + energy_today: + entity: sensor.sim_open_meteo_energy_production_today + unit: kWh + criteria: [domain, device_class, translation_key, platforms, units] + energy_tomorrow: + entity: sensor.sim_open_meteo_energy_production_tomorrow + unit: kWh + criteria: [domain, device_class, translation_key, platforms, units] + energy_d2: + none: true + energy_d3: + none: true diff --git a/tests/scenarios/forecast/solcast.pv.yaml b/tests/scenarios/forecast/solcast.pv.yaml index a1c4e7d..e178d55 100644 --- a/tests/scenarios/forecast/solcast.pv.yaml +++ b/tests/scenarios/forecast/solcast.pv.yaml @@ -9,3 +9,7 @@ roles: entity: sensor.sim_solcast_forecast_tomorrow unit: kWh criteria: [domain, device_class, translation_key, platforms, units] + forecast_d3: + none: true + forecast_d4: + none: true diff --git a/tests/scenarios/test_conformance.py b/tests/scenarios/test_conformance.py index 6de9f5c..6fb9edf 100644 --- a/tests/scenarios/test_conformance.py +++ b/tests/scenarios/test_conformance.py @@ -22,7 +22,7 @@ import asyncio import itertools import re -from collections.abc import AsyncIterator, Mapping +from collections.abc import AsyncIterator, Mapping, Sequence from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -35,7 +35,7 @@ from suncore.core.devices import DeviceConfig from suncore.core.entities import EntityMeta from suncore.core.site import EntityRef -from suncore.core.units import parse_unit +from suncore.core.units import Dimension, parse_unit from suncore.ha.connection import EntityState from suncore.sim.fake_ha import FakeHAConnection from suncore.sim.scenario import load_scenario @@ -49,7 +49,7 @@ matched_criteria, rank_entities, ) -from suncore.templates.models import AttributeSeries +from suncore.templates.models import AttributeSeries, EnergyDashboardSeries pytestmark = pytest.mark.simulator @@ -205,7 +205,7 @@ async def test_template_conforms(template_id: str, fake: FakeHAConnection) -> No elif isinstance(template, TariffTemplate): await _check_tariff(template, fake, resolved, states) elif isinstance(template, ForecastTemplate): - _check_forecast(template, resolved, states) + await _check_forecast(template, fake, resolved, states) elif isinstance(template, RegionTemplate): await _check_region(template, fake) @@ -254,39 +254,71 @@ async def _check_tariff( assert template.price_series.fields.value_unit == "{currency}/kWh" -def _check_forecast( +def _energy_kwh(points: Sequence[float], unit_text: str, resolution_min: int) -> float: + """Energy of equally spaced values: power = mean over the slot, energy = per slot.""" + unit = parse_unit(unit_text) + assert unit is not None + base = sum(points) * unit.factor # W or Wh + wh = base * resolution_min / 60 if unit.dimension is Dimension.POWER else base + return wh / 1000 + + +async def _check_forecast( template: ForecastTemplate, + fake: FakeHAConnection, resolved: Mapping[str, EntityMeta], states: Mapping[str, EntityState], ) -> None: series = template.series - assert isinstance(series, AttributeSeries) fields = series.fields - unit = parse_unit(fields.value_unit) - assert unit is not None - points: list[dict[str, Any]] = [] - for role in series.roles: + assert parse_unit(fields.value_unit) is not None + if isinstance(series, EnergyDashboardSeries): + entries = await fake.energy_solar_forecast() + assert entries + hours = entries["sim_entry_forecast_solar"] + stamps = sorted(datetime.fromisoformat(k) for k in hours) + assert all(s.utcoffset() is not None for s in stamps) + assert all(v >= 0 for v in hours.values()) + # The integration drops only a zero at 00:00:00: spacing is a multiple of the period. + assert { + (b - a).total_seconds() / 60 % template.resolution_min + for a, b in itertools.pairwise(stamps) + } == {0.0} + return + assert isinstance(series, AttributeSeries) + resolved_roles = [r for r in series.roles if r in resolved] + assert resolved_roles, "a forecast scenario resolves at least one series role" + for role in resolved_roles: entity = states[resolved[role].entity_id] attribute = entity.attributes[series.attribute] - assert isinstance(attribute, list) - assert attribute, role - points.extend(attribute) - required = {fields.timestamp_field, fields.value_field, fields.p10_field, fields.p90_field} - assert all(required <= set(p) for p in points) - stamps = [datetime.fromisoformat(p[fields.timestamp_field]) for p in points] - assert stamps == sorted(stamps) - deltas = {(b - a).total_seconds() / 60 for a, b in itertools.pairwise(stamps)} - assert deltas == {template.resolution_min} - assert all(p[fields.p10_field] <= p[fields.value_field] <= p[fields.p90_field] for p in points) - # kW is average power over the slot: the day sensor in kWh equals sum(kW) * slot hours. - today = states[resolved[series.roles[0]].entity_id] - day = [ - p - for p in points - if p[fields.timestamp_field][:10] == points[0][fields.timestamp_field][:10] - ] - kwh = sum(p[fields.value_field] for p in day) * template.resolution_min / 60 - assert float(today.state) == pytest.approx(kwh, abs=0.05) + if series.shape == "mapping": + assert isinstance(attribute, dict) + assert attribute, role + stamps = [datetime.fromisoformat(k) for k in attribute] + values = list(attribute.values()) + assert all(isinstance(v, int | float) and v >= 0 for v in values) + else: + assert isinstance(attribute, list) + assert attribute, role + required = {fields.timestamp_field, fields.value_field} + required |= {f for f in (fields.p10_field, fields.p90_field) if f is not None} + assert all(required <= set(p) for p in attribute) + stamps = [datetime.fromisoformat(p[fields.timestamp_field]) for p in attribute] + values = [p[fields.value_field] for p in attribute] + if fields.p10_field is not None and fields.p90_field is not None: + assert all( + p[fields.p10_field] <= p[fields.value_field] <= p[fields.p90_field] + for p in attribute + ) + assert all(s.utcoffset() is not None for s in stamps) # offsets, never naive times + assert stamps == sorted(stamps) + deltas = {(b - a).total_seconds() / 60 for a, b in itertools.pairwise(stamps)} + assert deltas == {template.resolution_min} + # The day sensor's state is the sum of its series (kWh). + state_unit = parse_unit(entity.attributes["unit_of_measurement"]) + assert state_unit is not None + expected = _energy_kwh(values, fields.value_unit, template.resolution_min) + assert float(entity.state) * state_unit.factor / 1000 == pytest.approx(expected, abs=0.05) async def _check_region(template: RegionTemplate, fake: FakeHAConnection) -> None: diff --git a/tests/sim/test_forecast_blend.py b/tests/sim/test_forecast_blend.py new file mode 100644 index 0000000..34ba816 --- /dev/null +++ b/tests/sim/test_forecast_blend.py @@ -0,0 +1,101 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The demo's three biased PV sources, blended by their measured skill (WP5 part B). + +Solcast-like (unbiased, 6 % noise), Open-Meteo-like (6 % low, 8 % noise) and Forecast.Solar-like +(4 % high, 7 % noise) forecasts of the simulator are issued at every local midnight for that day. +Fourteen days score them against the simulator's truth (clear sky x cloud factor); the fifteenth +day is blended with those scores. Deterministic: fixed forecast seed and cloud seed. +""" + +from __future__ import annotations + +import random +from datetime import datetime, timedelta + +import pytest + +from suncore.core.forecast import AccuracyRecord +from suncore.core.series import QuantileSeries, SeriesUnit, TimeSeries +from suncore.forecast.accuracy import quarter_errors +from suncore.forecast.pv.blend import WEIGHT_BUCKETS, BlendInput, blend +from suncore.sim.forecasts import FORECAST_SEED, SimForecastModel +from suncore.sim.house import clear_sky_w +from suncore.sim.scenario import load_scenario + +pytestmark = pytest.mark.simulator + +SCENARIO = load_scenario("summer_day") +FIRST = SCENARIO.start.replace(hour=0, minute=0, second=0, microsecond=0) +Q = timedelta(minutes=15) +CLOUD_SEED = 4711 +SOURCES = ("solcast", "open_meteo", "forecast_solar") + + +def cloud_at(moment: datetime) -> float: + """A cloud factor per local day (0.45-1.0), the same for every call.""" + return random.Random(f"{CLOUD_SEED}|{moment.date().isoformat()}").uniform(0.45, 1.0) + + +MODEL = SimForecastModel(SCENARIO, cloud_at, seed=FORECAST_SEED) + + +def series(start: datetime, step: timedelta, values: list[float]) -> TimeSeries: + return TimeSeries(start=start, step=step, values=tuple(values), unit=SeriesUnit.W) + + +def truth(midnight: datetime) -> TimeSeries: + values = [ + clear_sky_w(midnight + Q * i + Q / 2, SCENARIO.house.pv) * cloud_at(midnight + Q * i) + for i in range(96) + ] + return series(midnight, Q, values) + + +def issued(source: str, midnight: datetime) -> TimeSeries: + """What ``source`` said for the day at its midnight, in W on the source's own grid.""" + if source == "solcast": + slots = MODEL.solcast_day(midnight, midnight) + return series(midnight, timedelta(minutes=30), [s["pv_estimate"] * 1000 for s in slots]) + if source == "open_meteo": + wh = MODEL.open_meteo_day(midnight, midnight) + return series(midnight, Q, [v * 4 for v in wh.values()]) + hours = MODEL.forecast_solar_wh_hours(midnight, 24) + values = [hours.get((midnight + timedelta(hours=h)).isoformat(), 0.0) for h in range(24)] + return series(midnight, timedelta(hours=1), values) + + +def mae(forecast: TimeSeries, actual: TimeSeries) -> float: + errors = quarter_errors(forecast, actual) + return sum(abs(e) for e in errors) / len(errors) + + +def test_the_blend_beats_the_worst_source_and_weights_follow_the_skill() -> None: + history = [FIRST + timedelta(days=d) for d in range(14)] + scores = { + source: sum(mae(issued(source, day), truth(day)) for day in history) / len(history) + for source in SOURCES + } + inputs = [ + BlendInput( + source, + QuantileSeries.deterministic(issued(source, FIRST + timedelta(days=14))), + records=[ + AccuracyRecord(source_id=source, horizon=b, mae_w=scores[source], bias_w=0, n=1344) + for b in WEIGHT_BUCKETS + ], + ) + for source in SOURCES + ] + target = FIRST + timedelta(days=14) + result = blend(inputs, target, horizon=timedelta(hours=24)) + assert result is not None + actual = truth(target) + blended = mae(result.series.p50, actual) + single = {i.source_id: mae(i.series.p50, actual) for i in inputs} + assert blended < max(single.values()) + assert blended <= sorted(single.values())[1] # at least as good as the middle source + weights = result.weights[0].weights + best = min(scores, key=scores.__getitem__) + worst = max(scores, key=scores.__getitem__) + assert weights[best] > weights[worst] + assert sum(weights.values()) == pytest.approx(1.0) diff --git a/tests/sim/test_forecasts.py b/tests/sim/test_forecasts.py new file mode 100644 index 0000000..fa9ca7b --- /dev/null +++ b/tests/sim/test_forecasts.py @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +"""The demo's forecast shapes: deterministic, derived from the PV model, documented shapes.""" + +from __future__ import annotations + +from datetime import datetime, timedelta + +from suncore.sim.forecasts import SimForecastModel +from suncore.sim.scenario import load_scenario + +SCENARIO = load_scenario("summer_day") +NOW = SCENARIO.start.replace(hour=9, minute=0) +MIDNIGHT = NOW.replace(hour=0) + + +def model(seed: int = 1) -> SimForecastModel: + return SimForecastModel(SCENARIO, lambda _: 0.9, seed=seed) + + +def test_same_seed_same_numbers_other_seed_other_numbers() -> None: + a, b = model(1), model(1) + assert a.solcast_day(MIDNIGHT, NOW) == b.solcast_day(MIDNIGHT, NOW) + assert a.open_meteo_day(MIDNIGHT, NOW) == b.open_meteo_day(MIDNIGHT, NOW) + assert a.solcast_day(MIDNIGHT, NOW) != model(2).solcast_day(MIDNIGHT, NOW) + + +def test_solcast_shape_and_quantile_order() -> None: + slots = model().solcast_day(MIDNIGHT, NOW) + assert len(slots) == 48 + assert set(slots[0]) == {"period_start", "pv_estimate", "pv_estimate10", "pv_estimate90"} + assert datetime.fromisoformat(slots[1]["period_start"]) - datetime.fromisoformat( + slots[0]["period_start"] + ) == timedelta(minutes=30) + assert all(s["pv_estimate10"] <= s["pv_estimate"] <= s["pv_estimate90"] for s in slots) + assert slots[0]["pv_estimate"] == 0 + assert max(s["pv_estimate"] for s in slots) > 1 + + +def test_open_meteo_and_forecast_solar_shapes() -> None: + wh = model().open_meteo_day(MIDNIGHT, NOW) + assert len(wh) == 96 + assert all(v >= 0 for v in wh.values()) + hours = model().forecast_solar_wh_hours(NOW) + # only a zero at local midnight is dropped: hours are contiguous apart from midnight + assert not any(v == 0 and k.endswith("T00:00:00" + k[-6:]) for k, v in hours.items()) + assert any(v == 0 for v in hours.values()) # night zeros are listed + half = model().solcast_wh_hours(NOW, 48, NOW) + assert len(half) == 96 + points = model().template_points(MIDNIGHT, 2, NOW) + assert len(points) == 96 + assert set(points[0]) == {"start", "power_w"} diff --git a/tests/templates/test_models.py b/tests/templates/test_models.py index a8fd44b..82d4e9e 100644 --- a/tests/templates/test_models.py +++ b/tests/templates/test_models.py @@ -287,3 +287,31 @@ def test_attribute_series_mapping_shape() -> None: nested["series"]["list_path"] = "result.watts" assert TEMPLATE_ADAPTER.validate_python(nested).kind == "forecast" _bad(nested, lambda d: d["series"].update(list_path="result..watts")) + + +def test_energy_solar_forecast_series_and_fallback() -> None: + doc = forecast_doc() + doc["series"] = { + "source": "energy_solar_forecast", + "missing_is_zero": True, + "fields": {"value_unit": "Wh"}, + } + doc["roles"] = {} + template = TEMPLATE_ADAPTER.validate_python(doc) + assert template.series.missing_is_zero is True # type: ignore[union-attr] + doc["series"]["fields"]["timestamp_field"] = "t" + with pytest.raises(ValidationError): + TEMPLATE_ADAPTER.validate_python(doc) + doc2 = forecast_doc() + doc2["fallback_series"] = { + "source": "action", + "domain": "tibber", + "service": "get_prices", + "response_path": "data", + "fields": {"timestamp_field": "a", "value_field": "b", "value_unit": "kW"}, + "max_horizon_h": 24, + } + assert TEMPLATE_ADAPTER.validate_python(doc2).fallback_series is not None # type: ignore[union-attr] + doc2["fallback_series"]["service"] = "evil" + with pytest.raises(ValidationError): + TEMPLATE_ADAPTER.validate_python(doc2)