GB GridPulse

Great Britain electricity-demand, renewable-generation and system-frequency analytics

GB GridPulse is an end-to-end portfolio project that demonstrates how official National Energy System Operator (NESO) data can be downloaded, validated, normalised, processed with Spark, stored as partitioned Parquet/Hive tables, analysed with SQL/Python/R and prepared for BI reporting.

Portfolio evidence statement: the repository includes small synthetic sample files so every reviewer can run the project immediately. Metrics generated from those sample files are illustrative and must not be interpreted as real grid-performance results.

Decision problem

Energy and grid-operations teams need to monitor demand pressure, understand the contribution of embedded wind and solar, detect sustained frequency excursions and compare forecast methods before relying on a model operationally.

This project supports five questions:

1. When does national demand peak, and how does renewable contribution change alongside it?
2. Which settlement periods create the highest net-demand pressure?
3. How often does system frequency move outside the 49.5–50.5 Hz statutory band?
4. Are out-of-band seconds isolated points or part of sustained operational events?
5. Does a Random Forest benchmark improve on a simple lag-1 forecast?

What makes the project credible

• Accepts both the included sample schema and official NESO column names such as ND, TSD, EMBEDDED_WIND_GENERATION, EMBEDDED_SOLAR_GENERATION, dtm and f.
• Uses strict time-based model evaluation and compares the model with a naive lag-1 baseline.
• Groups consecutive out-of-band frequency seconds into event-level records.
• Aligns Spark output paths with the supplied HDFS and Hive definitions.
• Separates implemented artefacts from future BI build specifications, avoiding unsupported claims.
• Includes automated tests for schema mapping, frequency thresholds, event grouping and settlement-period validation.

Architecture

flowchart LR
    A[NESO CKAN CSV resources] --> B[Python downloader]
    B --> C[HDFS raw zone]
    C --> D[PySpark normalisation and feature engineering]
    D --> E[Partitioned Parquet gold zone]
    E --> F[Hive external tables]
    F --> G[SQL analytical marts]
    E --> H[Python forecast benchmark]
    E --> I[R statistical diagnostics]
    G --> J[Power BI / Tableau build specifications]
    H --> K[Excel management report]
    I --> K

Loading

Technologies and evidence

Technology	How it is used
Python	Downloading, validation, feature engineering, event detection and forecasting
pandas / NumPy	Local data transformation and sample pipeline
scikit-learn	Random Forest benchmark and evaluation against a lag-1 baseline
PySpark / Spark SQL	Distributed schema normalisation, derived features and partitioned Parquet output
Hadoop HDFS	Raw and curated storage commands
Apache Hive / HiveQL	External analytical tables and business queries
SQL	Peak-demand, net-demand and frequency-event analysis
R	Correlation, regression and time-series diagnostic charts
Jupyter	Reproducible exploratory analysis
Excel	Management report with separate demand and renewable-share charts
Power BI / DAX	Detailed dashboard build specification; no .pbix file is claimed
Tableau	Operational dashboard build specification; no workbook is claimed
pytest	Seven automated transformation and validation tests
Git / GitHub	Version control and portfolio delivery

Repository structure

GB_GridPulse/
├── data/
│   ├── raw/                 # official downloads (ignored by Git)
│   ├── processed/           # local Spark outputs (ignored by Git)
│   └── sample/              # portable synthetic sample data
├── dashboard/               # Power BI and Tableau build specifications
├── docs/                    # architecture, source mapping and run guides
├── notebooks/               # executed exploratory notebook
├── outputs/                 # generated runtime outputs (ignored by Git)
├── r/                       # statistical validation script
├── reports/                 # committed portfolio report and screenshots
├── sql/                     # HDFS, Hive and analytical SQL
├── src/                     # downloader, transformations, validation, Spark ETL and model
├── tests/                   # automated tests
├── run_demo.py
└── requirements.txt

Quick start

python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
python run_demo.py
pytest -q

The demo creates:

• outputs/data_quality_report.json
• outputs/daily_energy_summary.csv
• outputs/frequency_event_summary.csv
• outputs/forecast_metrics.csv
• outputs/forecast_results.csv
• outputs/feature_importance.csv
• outputs/daily_demand_trend.png
• outputs/forecast_comparison.png

Official data workflow

List matching NESO resources without downloading:

python src/download_data.py --years 2025 2026 --frequency-limit 1 --list-only

Download selected resources:

python src/download_data.py --years 2025 2026 --frequency-limit 1

Run Spark locally on the official download directory:

spark-submit src/spark_etl.py \
  --demand-input "data/raw/demand_*.csv" \
  --frequency-input "data/raw/frequency_*.csv" \
  --output-root data/processed

Run Spark against HDFS:

spark-submit src/spark_etl.py \
  --demand-input "hdfs:///projects/gb_gridpulse/raw/demand/*.csv" \
  --frequency-input "hdfs:///projects/gb_gridpulse/raw/frequency/*.csv" \
  --output-root "hdfs:///projects/gb_gridpulse/gold"

Then execute sql/hive_schema.sql and sql/business_queries.sql.

Sample result snapshot

The committed report was generated from the synthetic demonstration data. It proves that the pipeline executes; it does not measure real system performance.

Data sources

• NESO Historic Demand Data
• NESO System Frequency Data

NESO publishes system-frequency data at one-second resolution and describes the operational target as 50 Hz with a statutory limit of 0.5 Hz above or below.

Limitations and next steps

• The included sample is intentionally small and synthetic.
• The renewable-share metric is a proxy based on embedded wind and solar divided by national demand; it is not the full GB generation mix.
• A real Power BI .pbix or Tableau workbook is not included; the repository provides implementation-ready specifications.
• Production model assessment should use several years of official data, rolling time-series validation and seasonal error analysis.
• Full cluster execution evidence should be added when the pipeline is run on Hadoop/Spark infrastructure.

CV-ready description

Built an end-to-end GB electricity analytics pipeline using Python, PySpark, Hadoop, Hive and SQL, including official NESO schema normalisation, partitioned Parquet marts, frequency-event detection and a demand-forecast benchmark against a naive baseline.

Licence

Code is provided under the MIT License. Source datasets remain subject to the terms published by NESO; source links and field definitions are documented in docs/official_source_mapping.md.