Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ac0f407
wip(forecast): PV forecast adapters, work in progress (WP5)
claude Oct 19, 2026
82ea8f7
test(forecast): sim shapes, template models, runtime and API tests (WP5)
claude Oct 19, 2026
b5666de
fix(app): run the accuracy job in the background and stop it before t…
claude Oct 19, 2026
b45e9cb
fix(forecast): wake the refresh loop by cancelling its sleeper instea…
claude Oct 19, 2026
37ec4eb
fix(forecast): a stop is never swallowed by a simultaneous wake; keep…
claude Oct 19, 2026
30b8380
feat(forecast): add the PV forecast adapters and accuracy bookkeeping…
claude Oct 19, 2026
b175b5f
feat(forecast): blend PV sources by inverse MAE with a minimum weight…
claude Oct 19, 2026
897c8e3
feat(forecast): learn a bounded PV bias per hour, season and weather …
claude Oct 19, 2026
d43a4bd
feat(forecast): detect PV clipping at the export limit and at a repea…
claude Oct 19, 2026
c9342ef
feat(forecast): serve the calibrated PV blend and learn nightly from …
claude Oct 19, 2026
33928b1
feat(forecast): blend PV sources, learn the bias and detect clipping …
claude Oct 19, 2026
b22bd4e
docs: STATUS for PR 4-D (WP5 PV forecast)
claude Oct 19, 2026
26fd7e6
fix(forecast): Forecast.Solar drops only the zero hour at midnight, n…
claude Oct 21, 2026
726dfd0
fix(forecast): an off-grid energy key never fails the Forecast.Solar …
claude Oct 21, 2026
6579033
docs: the built-in PV fallback moves to Phase 6 with the price fallbacks
claude Oct 21, 2026
b0d59b2
fix(forecast): leave sources older than 12 h out of the blend while a…
claude Oct 21, 2026
c183d7c
fix(forecast): refresh the PV forecasts as soon as Home Assistant (re…
claude Oct 21, 2026
f025845
test(forecast): a stop during the nightly PV learning releases the de…
claude Oct 21, 2026
69a21f5
fix(forecast): read each issue once for all its accuracy buckets and …
claude Oct 21, 2026
64e2683
fix(forecast): mark the Solcast fallback date-time format as UNVERIFI…
claude Oct 21, 2026
a99a446
docs: get_config latitude/longitude verified from source for the clea…
claude Oct 21, 2026
aabb333
fix(forecast): document that a zero export limit is not used for clip…
claude Oct 21, 2026
31d7caf
fix(forecast): the daily-peak reference is the 95th percentile of sto…
claude Oct 21, 2026
1f9b176
fix(forecast): serve the source status as the FetchStatus enum; tidy …
claude Oct 21, 2026
b62ba33
fix(forecast): the last midnight-only zero rule in tests and research…
claude Oct 21, 2026
1c743c3
fix(forecast): score the forecast accuracy on daylight quarter hours …
claude Oct 21, 2026
c1947f1
test(frontend): the wizard walk-through keeps the demo's other foreca…
claude Oct 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 47 additions & 3 deletions docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@
Read this first in every session, then `CLAUDE.md` (once it exists) and the
ADRs relevant to your task. Keep it short; history lives in git.

**Last update:** 2026-10-17 · **Phase:** 4 in progress – PR 4-A contracts, PR 4-B storage; Phase 3 (3-A
`phase-3/a-safety`, 3-B `phase-3/b-controller`, 3-C `phase-3/c-ui`) implemented, waiting for the
maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/zealous-mayer-j5oda9`)
**Last update:** 2026-10-19 · **Phase:** 4 in progress – PR 4-A contracts, PR 4-B storage, PR 4-D PV
forecast; Phase 3 (3-A `phase-3/a-safety`, 3-B `phase-3/b-controller`, 3-C `phase-3/c-ui`)
implemented, waiting for the maintainer's shadow test · **Branch:** `phase-4/d-pv-forecast`
(stacked on `phase-4/b-storage`)

## Done

Expand Down Expand Up @@ -58,6 +59,44 @@ maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/
- Follow-ups: `size_report` is not on a diagnostics page yet; a too-new `timeseries.db` is only
logged (a UI hint follows in 4-H).

- **Phase 4 – PR 4-D (WP5, PV forecast; branch `phase-4/d-pv-forecast`, stacked on 4-B)**:
- Part A – adapters: template-driven sources (`suncore/forecast/pv/`) for Solcast
(`detailedForecast` attributes, fallback read-only action `solcast_solar.query_forecast_data`,
never the Solcast API), Forecast.Solar (core, `energy/solar_forecast`, `config_entry_id` when
several entries are linked), Open-Meteo Solar Forecast (`wh_period_15m`) and the generic
attribute template; pure parsing onto the source's native grid (DST-safe, unknown slots stay
`None`). Refresh service: every 30 min and on entity changes (debounced), keeps the last good
data, bounded gap filling, stores issues in `timeseries.db` hourly or on a material change
(≥ 15 %). Nightly accuracy bookkeeping (MAE and bias = actual − forecast per source and
horizon bucket 0–6/6–24/24–48 h over 14 days, daylight quarter hours only, clipped ones
excluded, `learned_param` `forecast_accuracy`).
`GET /api/v1/forecast/pv` and `/forecast/accuracy`; the demo publishes three deterministic
biased sources.
- Part B – blending, calibration, clipping: `blend.py` (weights per horizon bucket from the
inverse 14-day MAE, minimum weight 0.1, manual weights as shares, equal weights until every
auto source has ≥ 96 scored quarter hours; renormalised per slot; P50 weighted, P10/P90 from
the sources with quantiles widened by the sources' disagreement); `calibration.py`
(multiplicative factor per local hour × month-group season × weather class, bounded 0.5–1.5,
shrunk with `n/(n+16)`, learnt nightly per source from the day-start issue without clipped
slots, applied only after `min_history_days` = 30 days with data, stored in `forecast_bias`);
`clearsky.py` (weather class = clear-sky index of the forecast itself: sun position + Haurwitz
clear sky on the configured arrays with HA's `latitude`/`longitude` from `get_config`;
fallback without location or `kwp`: the day's forecast peak against the 95th percentile of
the stored daily peaks);
`clipping.py` (grid export at the export limit while the forecast is higher; an inverter
plateau derived from the data – no AC-rating field exists, none was invented); `learning.py`
(nightly order clipping → accuracy incl. `blended` → bias tables). The service blends on the
15-min grid for now → +48 h, stores the blend as source `blended`, and the API shows the
weights, the calibration state per source and the blend without and with the learnt
calibration. The demo site now declares its 8 kWp horizontal plant.
- Deviation: the built-in PV fallback (Open-Meteo API, question 5) is **not shipped**; it moves
to Phase 6 with the price fallbacks (Open-Meteo terms and live azimuth convention
`UNVERIFIED`, plan §"Deviations during implementation").
- Zero-export plants (`export_limit_w` 0 or 0 %): clipping by the export limit is not detected
(grid ≈ 0 W also without curtailment); only the inverter-plateau rule applies.
- Follow-ups: the weights come from the *uncalibrated* sources' MAE (a calibrated source is
under-weighted until a calibrated accuracy is scored); no UI yet (4-H).

- **Fix after the first live test (2026-10-16, branch `fix/device-freshness`)**: on the maintainer's
HA the SunSpec battery SoC (84 % for 25 min) and the `smartthinq_sensors` power sensors (5 W for a
day) never advance `last_reported` for an unchanged value, so SunCore marked them "stale" and the
Expand Down Expand Up @@ -548,6 +587,11 @@ maintainer's shadow test · **Branch:** `phase-4/b-storage` (stacked on `claude/
| Default sensor cadences (power sensor 60 s = the client resync, energy counter without `counter_update_s` 300 s); whether a quiet plug re-reports an unchanged value so that `last_reported` moves | `suncore/app/cadence.py`, `research/ha-websocket-api.md` | shadow-mode run on the maintainer's system; set `power_update_s`/`counter_update_s` where a sensor is slower (`power_update_s` is in the device editor since WP10) |
| Controller tuning (deficit delay/budget, downside spread of the smoothing, counter-only climate needs `nominal + span + hysteresis` free surplus to shift; battery share without a goal = battery first) | `suncore/controller/`, `tests/controller/test_climate_equivalence.py` | shadow-mode run on the maintainer's system (plan exit criterion) |
| evcc REST API paths | `research/competitors.md` | Phase 6, MQTT API is verified and sufficient |
| Solcast day sensors d3/d4: translation keys `total_kwh_forecast_d3`/`_d4` from the research notes; the date-time format the fallback action `query_forecast_data` accepts for `start_date_time`/`end_date_time` (SunCore sends ISO 8601 with offset) | `templates/forecast/solcast.pv.yaml`, `suncore/forecast/pv/adapter.py` | maintainer's HA with Solcast |
| Open-Meteo Solar Forecast: translation key assumed equal to the description key; CC BY 4.0 attribution still to be shown in the UI (4-H) | `templates/forecast/open_meteo_solar_forecast.pv.yaml` | a running integration; PR 4-H |
| Forecast.Solar `wh_hours`/`watt_hours_period` keys read as period *start* (vendor API unreachable) | `templates/forecast/forecast_solar.pv.yaml`, `suncore/forecast/pv/parsing.py` | live comparison with the integration's sensors |
| Supervisor proxy forwards `energy/solar_forecast` | `suncore/ha/` (`fetch_solar_forecast`) | first run on HA OS |
| PV calibration constants (classes kt 0.3/0.65, shrink k = 16, decay 0.98, factor 0.5–1.5) and the clipping thresholds (plateau ±0.3 % for ≥ 1 h on ≥ 2 days; export within max(3 %, 50 W)) are reasoned, not fitted to real data | `suncore/forecast/pv/` | 30+ days of the maintainer's data |
| Solve times on Raspberry Pi 4 | ADR-0003 | Phase 4 benchmark |

## Environment notes
Expand Down
19 changes: 13 additions & 6 deletions docs/plans/phase-4.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ Phase 4 is done when (roadmap Phase 4 milestones plus the concept):
1. **Forecast adapters:** Solcast (P10/P50/P90 from the attributes or the read-only query action,
never the Solcast API), Forecast.Solar (core, via `energy/solar_forecast`), Open-Meteo Solar
Forecast (HACS), a generic attribute adapter (mapping per template), and the built-in fallback
behind an explicit opt-in (question 5). Several sources are blended by measured skill; a bias
behind an explicit opt-in (question 5; deferred to Phase 6, see "Deviations during
implementation"). Several sources are blended by measured skill; a bias
calibration per hour of day × season × weather class (cloudiness) is learnt; clipping
(inverter limit, export limit) is recognised and excluded from the statistics; accuracy (MAE,
bias per source and horizon) is stored and shown.
Expand Down Expand Up @@ -150,10 +151,11 @@ region; fixtures from verified shapes; templates with `source` and a conformance
### WP5 – PV forecast · `device-adapters` (adapters), `optimizer-engineer` (Opus, blending/calibration)

`suncore/forecast/pv/`: Solcast, Forecast.Solar, Open-Meteo Solar Forecast, generic attribute,
built-in fallback (question 5). Blending by inverse MAE over 14 days with a minimum weight and a
manual override; bias calibration per hour × season × weather class (clear-sky index buckets
from the forecast itself), bounded and shrinking to 1 with little data, active after
`min_history_days`; clipping detection; accuracy in `timeseries.db`.
built-in fallback (question 5; deferred to Phase 6, see "Deviations during implementation").
Blending by inverse MAE over 14 days with a minimum weight and a manual override; bias
calibration per hour × season × weather class (clear-sky index buckets from the forecast
itself), bounded and shrinking to 1 with little data, active after `min_history_days`;
clipping detection; accuracy in `timeseries.db`.

### WP6 – Load forecast · `optimizer-engineer` (Opus)

Expand Down Expand Up @@ -400,7 +402,12 @@ shadow and assisted run is the acceptance test.
Phase 6).
- Open-Meteo API terms are known only from the repository README (free for open-source and
non-commercial use, fair use, 10 000 requests/day, CC BY 4.0 attribution); the separate terms
page was unreachable. The built-in PV fallback stays opt-in and off.
page was unreachable.
- The built-in PV fallback (Open-Meteo API) is **not shipped in Phase 4** either (PR 4-D): its
terms and the azimuth convention of a live response are `UNVERIFIED` (question 5 default). It
moves to Phase 6 together with the price fallbacks, opt-in and off as planned; Phase 4 reads PV
forecasts only from the integrations the user already runs (Solcast, Forecast.Solar,
Open-Meteo Solar Forecast, generic template).
- `BatteryAction` is part of the 4-A contracts but rejected by the gate until WP10
(ADR-0011 addendum); its semantics are proposed in ADR-0013.

Expand Down
7 changes: 4 additions & 3 deletions docs/research/ha-websocket-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ the connector takes the URL and token as parameters.
| `call_service` | `domain`, `service`, `service_data`, `target`, `return_response` | actuation; `return_response: true` for actions that return data (`weather.get_forecasts`, `tibber.get_prices`, `solcast_solar.query_forecast_data`, `nordpool.get_prices_for_date`) |
| `get_states` | – | initial snapshot |
| `get_services` | – | capability discovery (which actions exist) |
| `get_config` | – | unit system, time zone, currency, location, version |
| `get_config` | – | unit system, time zone, currency, location, version; `latitude`/`longitude` (floats, degrees) feed the PV clear-sky model of the forecast calibration – verified in `home-assistant/core` `5cfe9aad7cb428c2c2ebb449719ae81976780de1` `homeassistant/core_config.py` `Config.as_dict` (lines 706, 713) returned unchanged by `websocket_api/commands.py` `handle_get_config` (only `external_url` is removed for local-only users) |
| `config/entity_registry/list_for_display` (preferred), `config/entity_registry/list`, `config/device_registry/list`, `config/area_registry/list` | – | setup wizard auto-detection (platform, device, area, category, translation key); verified, shapes in [Registries](#registries) |
| `render_template` | `template` | last resort for integrations without actions |
| `energy/get_prefs` | – | pre-fill from the Energy dashboard |
Expand Down Expand Up @@ -454,8 +454,9 @@ PyPI wheels `forecast-solar 5.0.1`, `open-meteo-solar-forecast 0.1.32`.
with `begin <= timestamp < end` for "current hour"
(`forecast-solar 5.0.1` wheel, `forecast_solar/models.py` `_interval_value_sum`), i.e. it
treats the key as the *start* of
the period. Entries with value 0 are dropped except non-midnight ones
(`val != 0 or time != 00:00:00`). `UNVERIFIED`: whether the vendor API itself
the period. Only an entry with value 0 that starts exactly at 00:00:00 is
dropped; every other zero is listed (kept: `val != 0 or time != 00:00:00`).
`UNVERIFIED`: whether the vendor API itself
labels `watt_hours_period` keys as period start or end (api.forecast.solar
unreachable); the HA library's own usage implies start.
- `open_meteo_solar_forecast` (HACS, `energy.py`): `wh_period`, hourly, key =
Expand Down
3 changes: 3 additions & 0 deletions docs/user/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ a template only tested in the simulator has never run against real hardware.

| Id | Kind | Name | Integration / platforms | Capabilities | Tested with | Unverified |
|---|---|---|---|---|---|---|
| `forecast_solar.pv` | forecast | Forecast.Solar PV forecast (Energy dashboard) | any | `pv_curve` | not tested on hardware | 8 |
| `generic.pv_forecast` | forecast | Generic PV forecast (attribute of a template sensor) | `template` | `pv_curve` | not tested on hardware | 1 |
| `open_meteo_solar_forecast.pv` | forecast | Open-Meteo Solar Forecast | `open_meteo_solar_forecast` | `pv_curve` | not tested on hardware | 5 |
| `solcast.pv` | forecast | Solcast PV forecast | `solcast_solar` | `pv_curve`, `pv_quantiles` | not tested on hardware | 4 |

## Regions
Expand Down
5 changes: 4 additions & 1 deletion frontend/e2e/wizard.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,10 @@ test.describe('setup wizard', () => {
import: { kind: 'fixed', fixed_price: 0.32 },
export: { kind: 'fixed', fixed_price: 0.08 },
});
expect(await get('forecast')).toMatchObject({ sources: [{ template_id: 'solcast.pv' }] });
// The demo keeps its other forecast sources (Open-Meteo, Forecast.Solar); Solcast is checked.
expect(await get('forecast')).toMatchObject({
sources: expect.arrayContaining([expect.objectContaining({ template_id: 'solcast.pv' })]),
});
const devices = (await get('devices')).devices;
expect(devices).toHaveLength(1);
expect(devices[0]).toMatchObject({ template_id: 'generic.switch', priority: 0 });
Expand Down
Loading
Loading