Repository navigation
Conversation
- adopt the renamed `environmental-score` endpoint (was `green-score`) - rename WattNet -> Wattnet throughout, `scores` -> `green_scores` - read WATTNET_EMAIL from the environment instead of hardcoding it - add `GreenDirector.footprint_series()` to fetch a datacenter's carbon/water intensity over an arbitrary window (used by the energy accounting backfill)
Add a `TUE` column to var/datacenters.csv (grid-to-compute ratio, known so far only for the ifca sites) and let conf.py tolerate blank numeric cells so the other datacenters can leave it empty.
jaimeib
force-pushed
the
feat/energy-footprint-stats
branch
from
September 10, 2026 07:50
dcbd3f8 to
0dcfd31
Compare
|
Core of the energy-accounting feature, no wiring into the API yet: - `mimir.py`: minimal Grafana Mimir (Prometheus) HTTP client - `accounting/compute.py`: pure helpers (resolve allocations, build the PromQL, sanitize + integrate power into 15-min energy buckets, apply the Wattnet footprint, downsample) - `accounting/store.py`: file-based store under `$ACCOUNTING_PTH/energy` (one JSON accumulator per deployment + append-only .series.jsonl) - `accounting/sweep.py`: background sweep that incrementally integrates the un-consolidated Mimir window per deployment and per datacenter, keeps a running accumulator plus a self-healing 24h tail, and the live `pointer_ts -> now` top-up used on the read path - `accounting/__init__.py`: read API (`get_accumulated*`, `get_series`, `get_user_energy`, `get_datacenter_power`, `get_cluster_energy`), degrading to None/empty when disabled or data is missing CPU (scaphandre) and GPU (DCGM) power sources are independent, so a site that only monitors its GPUs still accounts the GPU energy. All values are normalized by the per-datacenter TUE. Config lives in the new `energy` section of etc/main.yaml.
Wire `accounting.run_sweep()` into a `@repeat_every` task started on app startup, next to `get_cluster_stats_bg`. No-op unless the `energy` feature is enabled and the MIMIR_USER / MIMIR_PASSWORD credentials (added to template.env) are set.
…points Additive, nothing in the existing responses changes: - `EnergyStats` / `EnergyTimeSeries` / `EnergyTimeSeriesPoint` schemas - deployment endpoints (modules, tools, batch, try_me) gain an `energy` key and an `energy` query param (on by default): accumulated `EnergyStats` on the list view, extended `EnergyTimeSeries` with `full_info=true` - `GET /stats/user` gains an `energy` block (the user's deployments summed) and still serves it if the CSV usage stats are unavailable - `GET /stats/cluster` gains a per-datacenter `energy` block (site power draw + accumulated energy/footprint + optional 15-min series); no platform-wide total, and instrumented datacenters are kept even when the VO filter leaves them without nodes - internal callers (create_deployment, green-affinity, try_me capacity check) pass `energy=False` to skip the extra work
Runs without the monitoring stack: pure helpers always, the sweep and read API against a faked Mimir/Nomad, and an optional live Mimir probe behind PAPI_TESTS_MIMIR. Covers the GPU-only site case (no scaphandre series) at both the deployment and datacenter level.
- docs/energy-accounting.md: how the sweep, store, footprint and read paths work, plus the config reference and failure modes - docs/energy-api.md: the `energy` block shape for dashboard integration - README: short pointer to both, under the stats section
jaimeib
force-pushed
the
feat/energy-footprint-stats
branch
from
September 10, 2026 07:51
a7368f8 to
520fb9c
Compare
The empirical TUE for ifca-ai4eosc/ifca-imagine was 2.03; the correct value is 2.73. Update the datacenters.csv value, the config default, and the docs example. Also stop test_energy.py from asserting against the real per-datacenter TUE in var/datacenters.csv (it now pins its own value), so editing that file no longer breaks the test suite.
The read API used to report energy: null for any deployment the background sweep had not reached yet, which for a fresh deployment could mean waiting up to sweep_seconds (15 min). Close that gap with two fallback layers ahead of the periodic sweep: - On-demand sweep (Layer 3): when a read finds no accumulator doc, it runs one process_single pass synchronously instead of waiting for the background thread. Applies to single-deployment and bulk/per-user reads. - Pure-Mimir fallback (Layer 4): when even that leaves nothing settled (a deployment younger than one series_bucket_seconds), build a transient, never-persisted doc and read Mimir directly, only surfacing a result once Mimir actually has a sample (never a fabricated zero). Also simplify the live top-up: Mimir data is immutable once ingested, so there was no reason to hold back a fresh sample behind topup_lag_seconds or drop it as "stale" in compute.instant_power. Both the persisted and the live-only path now read the same lag-free window and always report the actual last sample, so live_only_topup collapses into the same topup() used everywhere else. Fix live_only_doc always reporting complete: false even though its window start is pinned to the exact, unclamped submit time.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ns/uuid/owner in these warnings can originate from request path/auth parameters that are not validated as strict identifiers before reaching here; %r escapes embedded newlines instead of writing them raw, so they cannot be used to forge log lines.
The previous fix (%r instead of %s) doesn't register as sanitization to SonarCloud's taint tracker, so the log-injection finding stayed open. Strip CR/LF explicitly via a small helper before these values reach any log call.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds per-deployment energy consumption and environmental footprint (carbon and water, scope 2) stats to PAPI, for the dashboard. Fully additive: nothing in the existing API changes behaviour and no new endpoints are added.
energykey: accumulatedEnergyStatson the list view, extendedEnergyTimeSeries(15-min series) withfull_info=true.GET /stats/usergains anenergyblock (the user's deployments summed), served even when the CSV usage stats are unavailable.GET /stats/clustergains a per-datacenterenergyblock (site power draw plus accumulated energy/footprint plus an optional series). No platform-wide total, clients sum the datacenters they care about.@repeat_everybackground thread. If the feature is disabled or its inputs are missing,energyisnulland the rest of the API is identical.How it works
Power comes from Grafana Mimir (
scaph_process_power_consumption_microwattsfor CPU,DCGM_FI_DEV_POWER_USAGEfor GPU, 30s scrape). A background sweep integrates the un-consolidated window into 15-min energy buckets per deployment and per datacenter, applies the per-datacenter TUE and the Wattnet carbon/water intensity, and keeps a running accumulator plus a self-healing 24h tail in a small JSON store under$ACCOUNTING_PTH/energy. On the read path a short live top-up (pointer_ts -> now, cached 25s) keeps the counter moving every ~30s.CPU and GPU sources are independent, so a site that only monitors its GPUs still accounts the GPU energy. Best-effort throughout: no data means
energy: null(never zeros), partial data dropscoverage_ratiobelow1.0.Full design and internals in docs/energy-accounting.md; dashboard-integration reference (response shapes, polling) in docs/energy-api.md.
Config and deployment
energysection inetc/main.yaml(enabled, sweep cadence, data-quality filter,metrics_datacenters, downsample caps).MIMIR_USER/MIMIR_PASSWORD(andWATTNET_EMAIL, previously hardcoded), seetemplate.env.TUEcolumn invar/datacenters.csv(populated for the ifca sites, blank elsewhere, falls back toenergy.default_tue_factor).green-scoreendpoint to the renamedenvironmental-scoreone.Testing
tests/test_energy.pyruns without the monitoring stack: pure helpers always, the sweep and read API against a faked Mimir/Nomad, an optional live Mimir probe behindPAPI_TESTS_MIMIR. Covers the GPU-only site case at both the deployment and datacenter level.Commits
The branch is split into 8 reviewable commits: gitignore, wattnet refactor, datacenters TUE column, accounting engine, background thread, endpoint integration, tests, docs.