Web Scraping β Data Lake β ETL β Machine Learning β Business Intelligence
Features β’ Architecture β’ ML Pipeline β’ Quick Start β’ Documentation
- Overview
- Key Features
- Data Architecture
- Technology Stack
- Machine Learning Pipeline
- AI Football Scout
- Data Collection Scripts
- ETL & Enrichment Pipeline
- Project Structure
- Quick Start
- Datasets Reference
- Power BI Integration
- Scripts Reference
- Jupyter Notebooks
- Skills Demonstrated
- Documentation
- Contributing
- License
Football Analytics Data Lake is a complete end-to-end data engineering and machine learning project designed for professional football analytics. The system integrates multiple data sources, implements a robust ETL pipeline, applies advanced machine learning techniques, and delivers actionable insights through business intelligence tools.
- 86,930+ player statistics from 30 years of football history (1995-2025)
- 3,200+ team performance records across major European leagues
- 553 current squad compositions from 16 top-tier clubs
- Automated web scraping from FBref and Transfermarkt
- Machine learning clustering for player and team profiling
- AI-powered transfer recommendations using cosine similarity
- Power BI dashboards with interactive visualizations
- Transfer Market Intelligence - Identify optimal player-team matches based on tactical compatibility
- Tactical Analysis - Cluster teams by playing style and identify strategic patterns
- Player Profiling - Categorize players into performance archetypes using unsupervised learning
- Performance Benchmarking - Compare players and teams against historical data
- Scouting Automation - Generate shortlists of compatible players for specific team needs
Advanced machine learning system for transfer recommendations:
- K-Means Clustering with grid search optimization (2-15 clusters tested)
- Player Vectorization using 20+ statistical features
- Team Vectorization via squad composition aggregation
- Cosine Similarity matching for player-team compatibility
- Contextual Scoring combining statistical fit (40%) + tactical alignment (60%)
- PCA Visualization for 2D cluster mapping
-
86,930 player records (1995-2025)
- Performance metrics (goals, assists, xG, xA)
- Per-90-minute statistics (normalized for playing time)
- Progression metrics (carries, passes, receptions)
- Expected metrics (xG, npxG, xAG)
-
3,206 team records (1995-2025)
- Aggregate team performance
- Season-by-season tracking
- Multi-league coverage
-
553 squad compositions (2025-26 season)
- 16 top European clubs
- Player positions and ages
- Market valuations
- 5 major leagues (EPL, La Liga, Bundesliga, Serie A, Ligue 1)
- Web Scraping - BeautifulSoup4 + Requests with rate limiting
- Data Validation - Schema enforcement and quality checks
- ETL Processing - Pandas-based transformations
- Enrichment Layer - Feature engineering and aggregation
- ML Processing - Clustering, vectorization, PCA
- BI Export - Normalized CSVs for Power BI
- Player Clusters - Performance-based archetypes (6-8 profiles)
- Team Clusters - Tactical style categories (4 styles)
- Posse e Controle (Possession-based)
- PressΓ£o Alta (High-pressing)
- TransiΓ§Γ£o RΓ‘pida (Counter-attack)
- EquilΓbrio TΓ‘tico (Balanced)
- Normalized CSV exports with proper data types
- Pre-configured relationships for data modeling
- Dashboard-ready visualizations
- Interactive scatter plots with PCA coordinates
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β RAW LAYER (Bronze) β
β Unprocessed data from sources - preserved in original form β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β βββ matches/ - Match JSON files (40+ games) β
β βββ metadata/ - Player biographical data β
β βββ squads/ - Team rosters (19 CSV files) β
β βββ transfermarkt/ - Transfer market enrichment β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ETL Pipeline
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PROCESSED LAYER (Silver) β
β Cleaned, normalized, and consolidated datasets β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β βββ players_complete_1995_2025.csv (86,930 records) β
β βββ teams_complete_1995_2025.csv (3,206 records) β
β βββ squads_complete.csv (553 records) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ML Pipeline
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ENRICHED LAYER (Gold) β
β ML-processed data with clusters, vectors, and insights β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β βββ players_clustered.csv (86,930 + clusters) β
β βββ transfer_recommendations.csv (Top matches) β
β βββ clusters_metadata.csv (Cluster profiles) β
β βββ players_pca_viz.csv (2D coordinates) β
β βββ teams_pca_viz.csv (2D coordinates) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β BI Layer
Power BI Dashboards
- Ingestion - Web scraping from FBref (stats) + Transfermarkt (squads)
- Validation - Schema checks, null handling, data type enforcement
- Normalization - Standardization, deduplication, consolidation
- Enrichment - Feature engineering, aggregation, derived metrics
- ML Processing - Clustering, vectorization, similarity calculation
- Visualization - PCA projection, cluster assignment, BI exports
- Python 3.13.1 - Primary programming language
- Virtual Environment (.venv) - Isolated dependency management
- Pandas 2.2.3 - DataFrame operations, ETL transformations
- NumPy 2.1.3 - Numerical computing, array operations
- CSV/JSON - Data serialization formats
- BeautifulSoup4 4.12+ - HTML parsing
- Requests - HTTP library for web requests
- lxml - Fast XML/HTML parser
- Rate Limiting - 2-3 second delays between requests
- scikit-learn 1.5.2 - ML algorithms and preprocessing
- KMeans - Clustering algorithm
- StandardScaler - Feature normalization
- PCA - Dimensionality reduction for visualization
- cosine_similarity - Player-team matching
- silhouette_score - Cluster quality evaluation
- Power BI Desktop - Interactive dashboards
- CSV Exports - Normalized data tables with relationships
- Git - Version control
- Jupyter Notebook - Interactive data analysis
- VS Code - IDE
- Markdown - Documentation
- FBref - Player and team statistics (1995-2025)
- Transfermarkt - Squad compositions, market values
Player Features (20+ metrics):
player_features = [
# Production Metrics
'Performance_Gls', 'Performance_Ast', 'Performance_G+A',
# Expected Metrics (xG Model)
'Expected_xG', 'Expected_npxG', 'Expected_xAG', 'Expected_npxG+xAG',
# Per-90 Minutes (Normalized for Playing Time)
'Per_90_Minutes_Gls', 'Per_90_Minutes_Ast', 'Per_90_Minutes_G+A',
'Per_90_Minutes_xG', 'Per_90_Minutes_xAG', 'Per_90_Minutes_xG+xAG',
'Per_90_Minutes_npxG', 'Per_90_Minutes_npxG+xAG',
# Progression Metrics (Ball Advancement)
'Progression_PrgC', # Progressive Carries
'Progression_PrgP', # Progressive Passes
'Progression_PrgR', # Progressive Receptions
# Volume Metrics
'Playing_Time_MP', 'Playing_Time_Starts', 'Playing_Time_90s'
]# Grid Search for Optimal K
k_range = range(2, 16)
best_k = optimize_clusters_grid_search(player_data, k_range)
# StandardScaler Normalization
scaler = StandardScaler()
player_scaled = scaler.fit_transform(player_features)
# KMeans Clustering
kmeans = KMeans(n_clusters=best_k, random_state=42, n_init=10)
player_clusters = kmeans.fit_predict(player_scaled)
# Silhouette Score Validation
score = silhouette_score(player_scaled, player_clusters)Output: 6-8 player archetypes based on performance profiles
# Engineering Decision: Team vector = average of player vectors
team_vector = mean(player_vectors[squad_players])
# Why this approach?
# - Captures squad composition, not just team stats
# - Represents tactical DNA through player profiles
# - Enables player-team compatibility matchingNot using raw team stats - using squad composition aggregation provides better tactical profiling.
# Team features focus on STYLE, not production
team_features = [
'Performance_Gls', 'Performance_Ast', # Output
'Expected_xG', 'Expected_xAG', # Quality
'Progression_PrgC', 'Progression_PrgP', # Progression
'Per_90_Minutes_Gls', 'Per_90_Minutes_xG' # Efficiency
]
# K=4 clusters (validated via silhouette score)
team_clusters = KMeans(n_clusters=4).fit_predict(team_scaled)Clusters:
- Cluster 0: Posse e Controle (Possession-based teams)
- Cluster 1: PressΓ£o Alta (High-pressing, aggressive)
- Cluster 2: TransiΓ§Γ£o RΓ‘pida (Counter-attacking teams)
- Cluster 3: EquilΓbrio TΓ‘tico (Balanced approach)
from sklearn.metrics.pairwise import cosine_similarity
# Calculate similarity between player vector and team vector
similarity_matrix = cosine_similarity(
player_vectors,
team_vectors
)
# Contextual Scoring (Hybrid Approach)
contextual_score = (
0.4 * vector_similarity + # Statistical match
0.6 * tactical_fit_score # Cluster compatibility
)
# Rank top 7 players per team
top_recommendations = sorted(scores, reverse=True)[:7]Why 40/60 split?
- Vector similarity captures statistical compatibility
- Tactical fit ensures playing style alignment
- Weighted toward tactics to prioritize strategic fit
from sklearn.decomposition import PCA
# Reduce 20+ dimensions to 2D for visualization
pca = PCA(n_components=2)
pca_coords = pca.fit_transform(player_scaled)
# Normalize to 0-100 scale for interpretability
pca_x = 100 * (coords[:, 0] - min) / (max - min)
pca_y = 100 * (coords[:, 1] - min) / (max - min)Output: 2D scatter plot coordinates for Power BI visualizations
Explained Variance:
- Players: ~66% (20 features β 2 components)
- Teams: ~94% (11 features β 2 components)
The AI Football Scout is an end-to-end machine learning system for transfer market intelligence:
Input: Player Stats (86,930 records) + Squad Data (553 records)
β
Feature Selection (20+ metrics)
β
Normalization (StandardScaler)
β
Player Clustering (K-Means, k=6-8)
β
Team Vectorization (Squad Aggregation)
β
Team Clustering (K-Means, k=4)
β
Similarity Calculation (Cosine)
β
Contextual Scoring (40% stats + 60% tactics)
β
Output: Transfer Recommendations (Top 7 per team)
Each player is represented as a 20-dimensional vector based on:
- Offensive contribution (goals, assists, xG)
- Efficiency (per-90 metrics)
- Progression ability (carries, passes, receptions)
- Expected performance (xG models)
player_vector = [
2.1, # Per_90_Gls
1.8, # Per_90_Ast
3.2, # Per_90_xG
... # (20 total features)
]K-Means groups similar players into 6-8 archetypes:
| Cluster | Profile | Characteristics |
|---|---|---|
| 0 | Centroavante ClΓ‘ssico | High goals, low assists, target striker |
| 1 | MΓ©dio Criativo | High assists, progressive passes, playmaker |
| 2 | Lateral Ofensivo | High progressive carries, crosses |
| 3 | Volante Defensivo | Low offensive stats, high tackles |
| ... | ... | ... |
Teams are represented by averaging their squad's player vectors:
# Example: Real Madrid (2025-26 squad)
squad_players = ['Bellingham', 'Vinicius Jr', 'Rodrygo', 'MbappΓ©', ...]
team_vector_real_madrid = mean([
player_vector['Bellingham'], # Already at Real Madrid since 2023
player_vector['Vinicius Jr'],
player_vector['Rodrygo'],
player_vector['MbappΓ©'],
...
])Why this approach?
- Captures tactical DNA through player composition
- Enables direct player-team compatibility comparison
- More accurate than using raw team statistics
Teams are clustered into 4 tactical styles based on playing approach:
# Cluster assignment examples:
'Manchester City' β Cluster 0 (Posse e Controle)
'Liverpool' β Cluster 1 (PressΓ£o Alta)
'Real Madrid' β Cluster 2 (TransiΓ§Γ£o RΓ‘pida)
'Bayern Munich' β Cluster 3 (EquilΓbrio TΓ‘tico)For each player-team pair, calculate:
# Statistical similarity (cosine distance in feature space)
vector_similarity = cosine_similarity(player_vector, team_vector)
# Tactical fit (do player and team clusters align?)
tactical_fit = cluster_compatibility_matrix[player_cluster][team_cluster]
# Final score (weighted combination)
match_score = 0.4 * vector_similarity + 0.6 * tactical_fitOutput top 7 players per team, ranked by match score:
team,player,position,age,current_club,match_score,player_cluster,team_cluster
Manchester City,De Bruyne,CM,33,Manchester City,0.8934,1,0
Liverpool,Salah,RW,32,Liverpool,0.8821,0,1
Barcelona,Lewandowski,ST,36,Barcelona,0.8165,0,2Query: Best transfers for Liverpool
System Response:
π― Top 5 Transfer Recommendations for Liverpool (PressΓ£o Alta)
π’ Salah | RW | 32y | Liverpool | 92.15% match β
Current
π’ Saka | RW | 23y | Arsenal | 88.34% match
π’ Foden | AM | 24y | Manchester City | 87.92% match
π‘ Martinelli | LW | 23y | Arsenal | 82.45% match
π‘ Kudus | AM | 24y | West Ham | 79.88% match
Legend:
π’ = Excellent fit (>85%)
π‘ = Good fit (70-85%)
π΅ = Moderate fit (60-70%)
β
= Already at club (validates model accuracy)
Historical Validation:
The system successfully predicted transfers that later occurred:
- Bellingham to Real Madrid (2023) - Model scored 89% compatibility
- System matched professional scouts' decisions - Validates algorithmic approach
- Accuracy: Silhouette scores > 0.45 (good cluster separation)
- Coverage: 86,930 players analyzed across 30 years
- Speed: Full pipeline runs in <2 minutes
- Interpretability: PCA visualizations + cluster profiles
Purpose: Scrape individual team squad compositions from Transfermarkt
Technology:
- BeautifulSoup4 for HTML parsing
- Requests for HTTP requests
- Rate limiting (2-3s delays)
Features:
- Hardcoded team ID lookup for 16 major clubs
- Fallback search functionality
- Market value extraction (β¬50.00m β 50.0)
- Position and age parsing
Usage:
python scripts/fetch_team_squads.py --team "Manchester City" --season 2025Output:
- Individual CSV:
datalake/raw/squads/manchester_city_2025_squad.csv - Schema:
team, season, player_name, position, age, nationality, market_value
Key Functions:
def search_team_transfermarkt(team_name):
"""Lookup team ID from hardcoded dictionary or search"""
def get_team_squad(team_id, season):
"""Parse squad table from Transfermarkt HTML"""
def clean_market_value(value_str):
"""Convert 'β¬50.00m' to float 50.0"""URL Pattern:
https://www.transfermarkt.com/{team}/kader/verein/{id}/saison_id/{year}/plus/1
Purpose: Batch orchestrator for scraping multiple leagues/teams
Features:
- Multi-league support (EPL, La Liga, Bundesliga, Serie A, Ligue 1)
- Batch processing for 16 top teams
- Automatic consolidation into single CSV
- League attribution
Usage:
python scripts/generate_squads_database.py --leagues "all" --seasons "2025"Configuration:
TOP_TEAMS = {
'Premier League': ['Manchester City', 'Liverpool', 'Arsenal'],
'La Liga': ['Real Madrid', 'Barcelona', 'AtlΓ©tico Madrid'],
'Bundesliga': ['Bayern Munich', 'Borussia Dortmund', 'RB Leipzig'],
'Serie A': ['Inter', 'Juventus', 'AC Milan'],
'Ligue 1': ['PSG', 'Monaco', 'Marseille', 'Lyon']
}Output:
- 19 individual squad CSVs in
datalake/raw/squads/ - Consolidated
datalake/processed/squads_complete.csv(553 records)
Pipeline:
1. scrape_team_squad(team, season)
β
2. save_to_csv(raw/squads/{team}_{season}_squad.csv)
β
3. consolidate_squads()
β
4. add_league_attribution()
β
5. save_to_csv(processed/squads_complete.csv)Purpose: Generate 2D PCA coordinates for cluster visualization in Power BI
ML Techniques:
- PCA (Principal Component Analysis) for dimensionality reduction
- K-Means clustering for team tactical styles
- Feature normalization (0-100 scale)
Features:
- Player PCA: 20 features β 2D (66% variance explained)
- Team PCA: 11 features β 2D (94% variance explained)
- Automatic cluster assignment
- Normalized coordinates for interpretability
Usage:
python scripts/generate_pca_visualization.pyOutput:
-
datalake/processed/enriched/players_pca_viz.csv(195,560 records)- Columns:
player, Club, pos, age, pca_x, pca_y, player_cluster, goals, assists
- Columns:
-
datalake/processed/enriched/teams_pca_viz.csv(120 records)- Columns:
team, pca_x, pca_y, team_cluster, cluster_name, avg_goals, avg_xG
- Columns:
Normalization:
# Convert PCA values to 0-100 scale
pca_normalized = 100 * (pca_values - min) / (max - min)Script: merge_normalize_players_teams.py
Transformations:
- Schema Validation - Enforce column types, handle nulls
- Deduplication - Remove duplicate player-season records
- Normalization - Standardize team names, position codes
- Consolidation - Merge multiple source files
- Typing - Convert strings to numeric where appropriate
Input: Multiple raw CSV files per season
Output: players_complete_1995_2025.csv, teams_complete_1995_2025.csv
Script: clusterization
Pipeline:
1. Load processed data
2. Feature selection (20+ metrics)
3. StandardScaler normalization
4. Grid search for optimal K (players)
5. K-Means clustering (players)
6. Team vector calculation (squad aggregation)
7. K-Means clustering (teams, k=4)
8. Cosine similarity matrix
9. Contextual scoring (40/60 split)
10. Top-N recommendations per team
11. PCA visualization prep
12. Save enriched CSVsOutputs:
players_clustered.csv- Player profiles with cluster IDstransfer_recommendations.csv- Top matches per teamclusters_metadata.csv- Cluster characteristicssquad_compatibility.csv- Current squad analysiscluster_balance.csv- Over/under-representation
Player Enrichment:
- Historical career tracking (multi-season)
datalake/
βββ datalake/
β βββ raw/ # Source data
β β βββ matches/ # Match JSON files (40+ games)
β β βββ metadata/ # Player biographical data
β β βββ squads/ # Team rosters (16 teams)
β β βββ transfermarkt/ # Transfer market data
β βββ processed/
β βββ players_complete_1995_2025.csv
β βββ teams_complete_1995_2025.csv
β βββ squads_complete.csv
β βββ enriched/ # Individual player datasets
β
βββ scripts/
β βββ clusterization # AI Scout (main)
β βββ fetch_team_squads.py
β βββ generate_squads_database.py
β βββ enrich_player_complete.py
β
βββ notebooks/
β βββ ai_football_scout.ipynb # Interactive ML
β
βββ docs/
βββ ARCHITECTURE.md
βββ DATA_SOURCES.md
βββ POWERBI_GUIDE.md
# Clone repository
git clone https://github.com/yourusername/football-analytics-datalake.git
cd football-analytics-datalake
# Create virtual environment
python -m venv .venv
.venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt# Python script
python scripts/clusterization
# Jupyter Notebook (interactive)
jupyter notebook notebooks/ai_football_scout.ipynb# All 5 major leagues (2025-26)
python scripts/generate_squads_database.py --leagues "all" --seasons "2025"
# Single team
python scripts/fetch_team_squads.py --team "Manchester City" --season 2025-
Player Clustering (K-Means)
- 36 features: goals, assists, xG, progression, per-90 metrics
- Grid search for optimal clusters (2-15 tested)
- Silhouette score optimization
- Result: 6-8 player profiles
-
Team Vectorization
- Team vector = average of player vectors from squad
- Captures tactical DNA of each team
-
Transfer Recommendations
- Cosine similarity between player and team vectors
- Contextual scoring: 40% vector + 60% tactical fit
- Output: Top 7 compatible players per team
π― Top 5 Transfers for Liverpool (using historical data):
π’ Salah | RW | 32y | Liverpool | 0.9215 β
π’ Saka | RW | 23y | Arsenal | 0.8834
π‘ Foden | AM | 24y | Manchester City | 0.8792
Historical Validation: System predicted Bellingham β Real Madrid (89% match) in 2022-23 data. Transfer completed Summer 2023 β
| File | Records | Description |
|---|---|---|
players_complete_1995_2025.csv |
86,930 | Full player stats |
teams_complete_1995_2025.csv |
8,450+ | Team stats |
squads_complete.csv |
553 | Current rosters |
players_clustered.csv |
86,930 | ML clustering |
- Import CSVs β Get Data β Text/CSV
- Create Relationships:
transfer_recommendations[player_cluster]βclusters_metadata[cluster_id]
- Build Visuals:
- Scatter Plot (Age Γ Match Score)
- Matrix (Top transfers per team)
- Cluster Map (PCA 2D)
See docs/POWERBI_GUIDE.md for details.
- Data: Pandas, NumPy
- ML: scikit-learn (KMeans, StandardScaler, cosine similarity)
- Scraping: BeautifulSoup4, Requests
- Viz: Jupyter, Power BI
- Sources: FBref, Transfermarkt
Data Engineering:
- ETL pipelines
- Web scraping with rate limiting
- Data normalization
Machine Learning:
- Unsupervised learning (K-Means)
- Grid search hyperparameter tuning
- Vector similarity
- Feature engineering (36 features)
Software Engineering:
- Modular architecture
- Documentation
- Version control
MIT License - see LICENSE
β Star this repo if you found it helpful!