Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 39 additions & 10 deletions docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,41 @@ ADRs relevant to your task. Keep it short; history lives in git.

**Last update:** 2026-10-28 · **Phase:** 4 implemented in the stacked PRs 4-A (contracts), 4-B
(storage), 4-C (tariffs), 4-D (PV forecast), 4-E (load forecast and risk model), 4-F (planner and
runtime), 4-G (battery control) and 4-H (UI and docs, WP11 + WP13); only the optional 4-I (ML load
model) is open; waiting for the maintainer's acceptance on his system. Phase 3 (3-A `phase-3/a-safety`,
runtime), 4-G (battery control), 4-H (UI and docs, WP11 + WP13) and the optional 4-I (ML load
model, WP12); waiting for the maintainer's acceptance on his system. Phase 3 (3-A `phase-3/a-safety`,
3-B `phase-3/b-controller`, 3-C `phase-3/c-ui`) is implemented, waiting for the maintainer's shadow
test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
test · **Branch:** `phase-4/i-ml-load` (stacked on `phase-4/h-ui-docs`)

## Done

- **Phase 4 – PR 4-I, WP12 optional ML load model (branch `phase-4/i-ml-load`, stacked on 4-H;
`optimizer-engineer`)**:
- `suncore/forecast/load_ml.py` (NumPy only): KNN on analogue days - quarter-hour samples of
the base load history (`max(91, min_history_days)` days, ≥ 75 % coverage, unmeasured device
slots dropped after 7 clean days), features day type (holiday ≈ weekend), slot (±2), daily
mean temperature, 7-day load level, weak age term; K = 24, P50 weighted median, P10/P90
weighted quantiles, ≥ 0, monotone, deterministic (stable sort). Forecasts after 7 usable days.
- `forecast.learning.load_ml_mode`: `off` not computed; `shadow` (default) stored hourly with
the profiles' issue time as source `load_ml`, scored nightly like `load_profile`, shown, never
used; `auto` used by the planner (and the risk view) only with ≥ `min_history_days` days and a
fresh (≤ 36 h) paired 14-day comparison on the same issues and quarter hours (≥ 7 days) with
an MAE ≥ 2 % below the profiles' (left again only when no longer below: hysteresis), else the
profiles; fitted once a day in a worker thread, stale after a day without a fit; every choice
has a `forecast.load_model.*` reason, in `auto` also as a plan-level reason.
- API: `ForecastAccuracyView.load_model` and `LoadForecastView.model` (`LoadModelView`, additive);
`load_ml` rows in `/forecast/accuracy`. UI: "Machine-learning load model" in the forecast
panel (mode, model in use, reason, both MAEs, history days); mode select in wizard step 4.
en/de keys added (`check_reasons` green), user guide §2 "The machine-learning load model".
- Benchmark `scripts/bench_load_ml.py` (synthetic house, 48 h = 193 slots, median of 5, session
amd64 container): training + prediction 30 days 30 ms, 60 days 49 ms, 90 days 36 ms (budget
1 s; slow test `tests/forecast/test_load_ml_bench.py`). Whole service refit (SQLite read of 91
days, 3 devices, profile + KNN fit) 128 ms vs 82 ms without the KNN model; aarch64 not
measured (`UNVERIFIED`).
- Tests: `tests/forecast/test_load_ml.py` (features, DST, hypothesis: quantiles monotone,
non-negative, on the grid, deterministic, every `auto` rule), `test_load_ml_service.py`
(shadow/off/auto, fallback, window, nightly bookkeeping, stored scores), API, runtime, planner
pass-through, Vitest panel and form.

- **Phase 4 – PR 4-A (plan, WP1, WP2; branch `claude/zealous-mayer-j5oda9`)**:
- Plan `docs/plans/phase-4.md` **revision 2 approved** (2026-10-16), incl. the merged concept
"Prognose, Puffer und Akkuplanung" (§0) and the split into stacked PRs 4-A…4-I.
Expand Down Expand Up @@ -631,10 +659,10 @@ test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
runtime (WP8, WP9) → 4-G battery control (WP10) → 4-H frontend + docs (WP11, WP13); 4-I ML load
(WP12, optional, after 4-E). Merge only after the Phase 3 acceptance.
- **WP10, WP11 and WP13 done** (4-G and 4-H, see Done). All Phase 4 PRs are open and stacked; merge
order **#25 → #26 → #27/#29 → #30 → #31 → #33 → 4-H** (merge commits are fine; **squash is
order **#25 → #26 → #27/#29 → #30 → #31 → #33 → 4-H → 4-I** (merge commits are fine; **squash is
recommended**, one commit per PR on `main`), **only after the Phase 3 acceptance** (plan §6).
ADR-0013 accepted by the maintainer on 2026-10-19. Open: 4-I (optional ML load model), and the
follow-ups below.
ADR-0013 accepted by the maintainer on 2026-10-19. 4-I (optional ML load model) is implemented on
`phase-4/i-ml-load`; open are the follow-ups below.
- **Safety note:** the documented fallback-timer measurement lifts the assisted cap (via
`fallback_measured`); the procedure gets a `control-safety` review (the lead runs it). It needs
the app's Watchdog switch off, otherwise the Supervisor restarts the killed app and SunCore itself
Expand All @@ -649,7 +677,8 @@ test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
- Follow-ups from 4-H and the docs: `fallback_timeout_s` changes should drop the battery mode to
shadow on the server (follows); a switch for `battery.fallback_measured` in the panel (today:
export, edit, import), a hint for a too-new `timeseries.db` (the API exposes no such state), the
panel for editing `action_params`/`series_key`/`learning`.
panel for editing `action_params`/`series_key`/`learning.min_history_days` (the ML mode has a
select since 4-I).
- Maintainer questions from 4-G (PR description): grid charging on the Kostal (generic
force-charge entity or a §3.4 ADR); the fallback timer measurement procedure for
`battery.fallback_measured`. Answered: no permanent reserve floor on the inverter.
Expand Down Expand Up @@ -864,7 +893,7 @@ test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
4. Optional, later: enable external battery management via Modbus on the inverter, add the modbus
hub, and measure the fallback timer with the documented procedure; then set
`battery.fallback_measured`.
3. Merge the stacked PRs in order **#25 → #26 → #27/#29 → #30 → #31 → #33 → 4-H** (squash
3. Merge the stacked PRs in order **#25 → #26 → #27/#29 → #30 → #31 → #33 → 4-H → 4-I** (squash
recommended), after the Phase 3 acceptance; each rebase onto `main` after the previous merge.
Release the next release-please PR for the stable app after acceptance (schema 4: release note
"export before updating").
Expand Down Expand Up @@ -892,7 +921,7 @@ test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
| Publication time of next-day prices (Tibber, Nord Pool, ENTSO-E, EPEX) | `research/integrations.md` | observe on the maintainer's HA |
| Nord Pool `config_entry` id source, sensor translation keys, publication time; Octopus/ENTSO-E/EPEX entity matchers (platform and unit only, no translation keys); `start`/`end` sent to `tibber.get_prices` as ISO with offset; negative-price rule proxy for all-in sources | `templates/tariff/*.yaml`, `suncore/tariff/pricing.py` | live check on a system with the integration |
| VAT applied to negative spot prices | `suncore/core/tariff.py` | maintainer's bill / supplier terms |
| KNN load model performance on aarch64 | plan WP12 | CI arm runner benchmark |
| KNN load model run time on aarch64 (amd64: training + 48 h prediction 30–50 ms for 30–90 days, `scripts/bench_load_ml.py`); its constants (K 24, slot window ±2, scales 4 K / 15 % level / 120 days age, 2 % margin) are reasoned, not fitted | plan WP12, `suncore/forecast/load_ml.py` | CI arm runner or a Pi; 30+ days of the maintainer's data in `shadow` |
| Energy colour variable names (`--energy-*`) | `frontend/src/app.css` | read HA frontend source |
| HA UI menu paths in the user docs (Settings → Apps → App store, Repositories, Show in sidebar; German labels) | `addon/*/DOCS.md`, `README.md`, `docs/user/*/quick-start.md` | maintainer checks on HA OS during the beta install |
| Supervisor Core proxy passes `supported_features` through | `suncore/ha/websocket.py` | first run on HA OS |
Expand All @@ -918,7 +947,7 @@ test · **Branch:** `phase-4/h-ui-docs` (stacked on `phase-4/g-battery`)
| `weather.get_forecasts` `hourly` support of the maintainer's weather integration (the `temperature_unit` attribute and the conversion of the forecast to it are verified from source, `research/ha-websocket-api.md`) | `suncore/app/adapters.py` (`HaLoadReader`) | maintainer's HA |
| `workday.check_date` over the Supervisor proxy and with the maintainer's Workday configuration (the `days_offset` handling is verified from source; a changed country/holiday list is not visible in the attributes, so the day-type cache only resets on a changed entity, `workdays`, `excludes` or offset) | `suncore/app/adapters.py`, `suncore/forecast/load_service.py` | maintainer maps a Workday sensor |
| Load and risk constants (half-life 21 days, outlier z 3, shrink 3 days, 7 clean days, degree-day bases 15/22 °C, coverage 75 %/90 %, 20 h temperature days, anchors sunrise + 1 h / sunset − 1 h, sunrise threshold 2 %/20 W) and the default PV bands per weather class (clear 0.85/1.10, mixed 0.55/1.35, overcast 0.35/1.70) are reasoned, not fitted | `suncore/forecast/load.py`, `suncore/forecast/risk.py`, `risk_learning.py` | 30+ days of the maintainer's data |
| Known limitation (n6): the load history and the risk samples are not re-derived when the device list changes - a device added later is only subtracted from the base load from the day its power is recorded (or rebuilt), and older risk samples keep the old base load | `suncore/forecast/load_service.py`, `risk_learning.py` | accepted for Phase 4; revisit with 4-I |
| Known limitation (n6): the load history and the risk samples are not re-derived when the device list changes - a device added later is only subtracted from the base load from the day its power is recorded (or rebuilt), and older risk samples keep the old base load | `suncore/forecast/load_service.py`, `risk_learning.py` | accepted for Phase 4; the KNN model (4-I) shares the base load and the limitation; revisit in Phase 5 |

## Environment notes

Expand Down
40 changes: 40 additions & 0 deletions docs/plans/phase-4.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,9 @@ NumPy KNN on analogue days (features: day type, slot, temperature, recent load);
by default: scored next to the profiles, shown in the accuracy view, used only in `auto` while
better over the last 14 days. Benchmark on aarch64 (training + prediction < 1 s).

**Implemented in PR 4-I** (branch `phase-4/i-ml-load`); deviations see "Deviations during
implementation".

### WP13 – Docs, translations, STATUS · `docs-writer`, `i18n-sync` (Haiku)

User guide `docs/user/<lang>/planning-and-battery.md` (forecasts, risk level and buffers,
Expand Down Expand Up @@ -636,6 +639,43 @@ shadow and assisted run is the acceptance test.
- The battery card shows what the control state contains; there is no separate "release
pending" field in `BatteryControlState`, so that state is only visible through the reasons.

- PR 4-I, WP12 (optional ML load model; `optimizer-engineer`):
- `suncore/forecast/load_ml.py`: KNN on quarter-hour samples of analogue days; features day
type (holiday close to weekend, far from workday), wall-clock slot (candidates ±2 slots),
daily mean temperature (only when the forecast day has one), level = mean base load of the
**7 days** before the day (a whole week, so the level is free of the weekday pattern; the
query uses the 7 days before the fit), plus a weak age term that also breaks ties. K = 24,
weights `1 / (0.25 + distance)`, P50 = weighted **median** (the profiles use a weighted mean;
the median suits the MAE score), P10/P90 = weighted 10/90 % quantiles of the neighbours. The
constants are reasoned, not fitted (`UNVERIFIED` like the profile constants).
- History window of the KNN model: `max(91 days, min_history_days)` (the profiles keep their
56 days; the service reads the longer window once and slices it for the profiles). The fit
window now starts at a local midnight, so a DST change inside it no longer drops its first
day (also for the profiles).
- "Better" in `auto` = a **paired** 14-day score (review M2): the KNN forecast is stored with
the same issue time as the profile forecast, and the nightly job scores both on the common
issues and only on quarter hours where both and the measurement are known (pooled over the
horizon buckets, ≥ 7 × 96 quarter hours); it is kept in `learned_param` `ml_state/
load_comparison` with its time and counts only while ≤ 36 h old (review M1; a failed nightly
comparison drops it, a missing `load_ml` score is removed from the display). The KNN model is
entered at an MAE ≥ **2 %** below the profiles' and left only when it is no longer below
(hysteresis, review m4), with at least `min_history_days` usable days; otherwise the
profiles, each with a `forecast.load_model.*` reason. The per-source rows stay for display.
- The KNN model is fitted once per local day (or after `request_refit`) in a worker thread
(`asyncio.to_thread`); its forecast level is the 7 days before local midnight like every
sample's; a model fitted before yesterday (its fit keeps failing) is stale and not used
(review m1, m3, m4). The KNN forecast is computed only when it is stored or in use (m2).
- In `auto` the plan carries the choice as a plan-level reason (`build_inputs(load_reasons=…)`,
planner still imports only `core`); in `off`/`shadow` no plan reason (nothing to explain).
The risk view and the planner get the same selected forecast; the risk model's *learnt*
errors keep coming from the stored `load_profile` forecasts.
- The accuracy view shows `load_ml` as its own source row and `load_model` (mode, model in use,
reason, both pooled MAEs, history days) under "Machine-learning load model"; the forecast
wizard step got the mode select (`forecast.learning.load_ml_mode`, the only learning field
in the UI).
- Benchmark `scripts/bench_load_ml.py` (sibling of `bench_planner.py`), slow test < 1 s on CI;
aarch64 still `UNVERIFIED` (no arm runner job exists yet).

## 7. Out of scope for Phase 4

Full plan chart with device Gantt, zoom/pan, history, savings vs baselines, what-if and
Expand Down
33 changes: 31 additions & 2 deletions docs/user/de/planning-and-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,37 @@ Zwei optionale Eingaben in Schritt 4 (**Eingaben der Lastprognose**) verbessern
zeichnet SunCore deren Außentemperatur selbst auf. Der Temperaturterm der Lastprognose wird
nach 14 Tagen mit aufgezeichneter Temperatur aktiv.

Ein Lastmodell mit maschinellem Lernen ist **noch nicht verfügbar**; das Feld sagt es. Die Profile
funktionieren vom ersten Tag an.
Die Profile funktionieren vom ersten Tag an.

### Das maschinelle Lastmodell

Neben den Profilen rechnet SunCore ein optionales **maschinelles Lastmodell**. Es sucht die
vergangenen Tage, die dem kommenden am ähnlichsten sind (gleiche Tagesart – ein Feiertag zählt wie
ein Wochenendtag –, gleiche Uhrzeit, ähnliche Außentemperatur, wenn eine Wetter-Entität zugeordnet
ist, und ein ähnliches Lastniveau in der letzten Woche), und sagt jede Viertelstunde aus dem
Verbrauch an diesen Tagen voraus, mit einem Band aus deren Streuung. Es braucht keine zusätzliche
Software und rechnet in wenigen zehn Millisekunden.

Den Modus wählst du in Schritt 4 unter **Maschinelles Lastmodell**:

- **Aus**: Es wird nicht berechnet.
- **Nur Vergleich** (Standard): Es wird berechnet, gespeichert und jede Nacht genau wie die Profile
bewertet und in **Prognosen und Genauigkeit** unter *Maschinelles Lastmodell* angezeigt, beide
Fehler nebeneinander. Der Plan nutzt es nie.
- **Automatisch**: Der Plan nutzt es nur, solange zweierlei gilt: Es hat mindestens den Verlauf,
den die Lerneinstellung verlangt (standardmäßig 30 Tage), **und** im Vergleich der letzten Nacht
lag sein mittlerer absoluter Fehler über 14 Tage mindestens 2 % unter dem der Profile. Beide
Modelle werden an genau denselben Prognosen und Viertelstunden verglichen (mindestens eine
Woche davon). Ist es einmal in Gebrauch, bleibt es, bis es gar nicht mehr besser ist, damit es
nicht hin und her springt. Ohne frischen Vergleich (der nächtliche Lauf ist fehlgeschlagen)
oder mit einem Modell, das seit gestern nicht neu angelernt werden konnte, bleibt der Plan bei
den Profilen. Jeder Plan trägt dann einen kurzen Hinweis, welches Modell er genutzt hat und
warum („noch zu wenig Verlauf“, „nicht deutlich besser“, „genauer als die Profile“ mit beiden
Fehlern).

Das Modell wird einmal am Tag neu angelernt, der Vergleich entsteht im nächtlichen Lauf, ein
Wechsel zwischen den Modellen passiert also höchstens einmal am Tag. Das Risikomodell lernt seine
Fehlerverteilung weiter aus der Profilprognose.

## 3. Tarife

Expand Down
32 changes: 30 additions & 2 deletions docs/user/en/planning-and-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,36 @@ Two optional inputs in step 4 (**Load forecast inputs**) improve it:
records its outdoor temperature itself. The temperature term of the load forecast becomes
active after 14 days with a recorded temperature.

A machine-learning load model is **not available yet**; the panel says so. The profiles work from
day one.
The profiles work from day one.

### The machine-learning load model

Next to the profiles SunCore runs an optional **machine-learning load model**. It looks for the
past days that resemble the coming one most (same kind of day – a holiday counts like a weekend
day –, the same time of day, a similar outdoor temperature if a weather entity is mapped, and a
similar load level over the past week) and forecasts each quarter hour from what the house used
on those days, with a band from their spread. It needs no extra software and runs in a few tens of
milliseconds.

Choose its mode in step 4 under **Machine-learning load model**:

- **Off**: it is not computed.
- **Comparison only** (default): it is computed, stored and scored every night exactly like the
profiles, and shown in **Forecasts and accuracy** under *Machine-learning load model*, with both
errors side by side. The plan never uses it.
- **Automatic**: the plan uses it only while two things hold: it has at least the history the
learning setting asks for (30 days by default), **and** in last night's comparison its mean
absolute error over the last 14 days was at least 2 % lower than the profiles'. Both models are
compared on exactly the same forecasts and quarter hours (at least a week of them). Once in use,
the model stays until it is no longer better at all, so it does not flip back and forth. Without
a fresh comparison (the nightly job failed) or with a model that could not be refitted since
yesterday, the plan stays on the profiles. Each plan then carries a short note which model it
used and why ("not enough history yet", "not clearly better", "more accurate than the profiles"
with both errors).

The model is refitted once a day, the comparison is computed in the nightly job, so a switch
between the models happens at most once a day. The risk model keeps learning its error
distribution from the profile forecast.

## 3. Tariffs

Expand Down
3 changes: 2 additions & 1 deletion frontend/e2e/plan.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ test.describe('Plan view', () => {
await expect(panel.getByTestId('fc-accuracy')).toBeVisible();
await expect(panel.getByTestId('fc-weights')).toContainText('equal');
await expect(panel.getByTestId('fc-calibration')).toContainText('No history yet');
await expect(panel.getByTestId('fc-ml')).toContainText('Not available yet');
await expect(panel.getByTestId('fc-ml-mode')).toContainText('comparison only');
await expect(panel.getByTestId('fc-ml-in-use')).toContainText('the profiles');
const credit = panel.getByTestId('open-meteo-attribution');
await expect(credit).toContainText('CC BY 4.0');
await expect(credit.getByRole('link', { name: 'Open-Meteo.com' })).toHaveAttribute(
Expand Down
Loading
Loading