Telecom Data Generation Engine

Synthetic data engine for a mobile telecom operator.
It generates a small, consistent “mini data warehouse” of dimensions and facts: customers, segments, tariffs, locations, time, network KPIs, usage, and NOC tickets.

The code is designed to be:

• Config‑driven (central settings in config/settings.py)
• Reproducible (fixed random seeds)
• Modular (separate dimension and fact generators)
• Easy to extend (add new dimensions/facts without breaking the pipeline)

---

1. Environment and Dependencies

Python version

Tested with:

• Python 3.11 (recommended)
• Should also work with Python 3.10+

Installing dependencies

The project uses pyproject.toml. From the project root:

# create and activate a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate     # Linux / macOS
# or:
.\.venv\Scripts\activate      # Windows PowerShell

# install dependencies
pip install -e .

Key libraries used:

• pandas – tabular data manipulation
• numpy – random generation, numerical helpers

(Any other packages listed in pyproject.toml are installed automatically.)

---

2. Configuration and Default Values

All global “knobs” live in config/settings.py:

• Simulation horizon:
  • SIM_START_DATE
  • SIM_END_DATE
• Random seeds:
  • NETWORK_RANDOM_SEED for network KPIs
  • NOC_RANDOM_SEED for NOC tickets
• Network KPI thresholds:
  • BUSY_MIN_DL_MBPS, BUSY_MIN_RRC_SR_PCT, BUSY_MAX_DROP_PCT
  • NB_MIN_DL_MBPS, NB_MIN_RRC_SR_PCT, NB_MAX_DROP_PCT
• NOC logic:
  • NOC_BASE_ISSUE_PROB (random issue probability per hour)

Each module also has a dataclass config with sensible defaults:

• NetworkConfig (telecom_core/facts/network_kpi.py)
• SegmentUsageConfig (telecom_core/facts/segment_usage.py)
• NocConfig (telecom_core/facts/noc_ticket.py)
• TimeConfig (telecom_core/dimensions/time.py)
• TariffConfig (telecom_core/dimensions/tariff.py)
• CustomerConfig (telecom_core/dimensions/customer.py)

You can either rely on the defaults or pass your own instances when calling the generators.

---

3. Project Structure

data_engine/
├── config/
│   ├── __init__.py
│   └── settings.py          # global simulation settings (dates, seeds, thresholds)
│
├── telecom_core/
│   ├── __init__.py
│   ├── main.py              # orchestrates the full pipeline
│   │
│   ├── dimensions/
│   │   ├── __init__.py
│   │   ├── segment.py       # dim_customer_segment / segment_id
│   │   ├── location.py      # dim_location / location_id
│   │   ├── time.py          # dim_time / time_id
│   │   ├── tariff.py        # dim_tariff / tariff_id
│   │   └── customer.py      # dim_customer / customer_key
│   │
│   └── facts/
│       ├── __init__.py
│       ├── network_kpi.py   # fact_network_kpi_hourly
│       ├── segment_usage.py # fact_segment_usage_hourly
│       ├── network_noc_daily.py  # (optional aggregation, not in main pipeline)
│       └── noc_ticket.py    # fact_noc_ticket
│
├── output/                  # generated CSV files (created at runtime)
└── pyproject.toml

All ID columns follow a consistent naming convention: xxxxx_id
(e.g. segment_id, location_id, time_id, tariff_id, customer_key).

---

4. Running the Pipeline

From the project root (where telecom_core/ lives):

# inside your virtual environment
python -m telecom_core.main

This will:

1. Create an output/ directory if it does not exist.
2. Generate all dimensions and facts.
3. Save them as CSV files in output/.
4. Print detailed logs showing each step and row counts, e.g.:

[STEP 1] Building dimension tables...
  - Generating dim_customer_segment ...
  - Generating dim_location ...
  - Generating dim_time ...
  - Generating dim_tariff ...
  - Generating dim_customer ...
[STEP 2] Generating fact_network_kpi_hourly ...
[STEP 3] Preparing SegmentUsageConfig ...
[STEP 4] Generating fact_segment_usage_hourly ...
[STEP 5] Generating fact_noc_ticket ...
Generation complete. CSV files written to: .../output

---

5. Generated Tables and Logic

5.1 Dimensions

dim_customer_segment (telecom_core/dimensions/segment.py)

• Function: generate_dim_segment(config: SegmentConfig | None = None)
• Grain: one row per segment (High, Normal, Low)
• Key columns:
  • segment_id (1=High, 2=Normal, 3=Low)
  • segment_name
  • global_target_share (target customer mix)
  • revenue_contribution
  • avg_monthly_revenue
  • data_usage_pattern (Heavy/Moderate/Light)
  • complaint_likelihood
  • sensitivity_to_outage
• Most values are hard-coded business assumptions, but the target shares are controlled by SegmentConfig.

dim_location (telecom_core/dimensions/location.py)

• Function: generate_dim_location(config: LocationConfig | None = None)
• Grain: one row per location_id (20 locations)
• Key columns:
  • location_id
  • city (e.g. “Düsseldorf”)
  • location_name
  • location_type (Commercial, Residential, Industrial, Transport)
  • revenue_potential (High, Medium, Low)
  • population_density
  • coverage_quality
• Locations are hard-coded; LocationConfig controls global attributes like default city.

dim_time (telecom_core/dimensions/time.py)

• Function: generate_dim_time(config: TimeConfig) -> pd.DataFrame
• Grain: one row per hour between start_date and end_date (exclusive)
• Key columns:
  • time_id (YYYYMMDDHH integer)
  • date
  • hour
  • is_busy_hour
  • is_working_hour
  • is_weekend
  • is_holiday
• Busy and working hours are rule‑based.
• Dates are read from TimeConfig, which in main.py is built from settings.SIM_START_DATE and settings.SIM_END_DATE.

dim_tariff (telecom_core/dimensions/tariff.py)

• Function: build_dim_tariff(config: TariffConfig | None = None)
• Grain: one row per tariff_id
• Represents 4 tariffs with increasing quota, speed, and price.
• Key columns:
  • tariff_id, tariff_name, segment_target
  • data_quota_gb, price_eur
  • dl_speed_mbps, ul_speed_mbps
  • network_tech (default '4G')
• Values are hard‑coded inside the function; TariffConfig mostly holds shared attributes (tech).

dim_customer (telecom_core/dimensions/customer.py)

• Function:
  build_dim_customer(dim_location, dim_segment, dim_tariff, config: CustomerConfig | None)
• Grain: one row per synthetic customer
• Key columns:
  • customer_key (surrogate key)
  • segment_id
  • location_id
  • tariff_id
  • signup_date
  • status
• Logic:
  • Uses CustomerConfig.num_customers, signup_start_date, signup_end_date, random_seed.
  • Segment mix is driven by dim_segment.global_target_share (so changing SegmentConfig propagates to customers).
  • Location distribution is biased by population_density and revenue_potential (different patterns for each segment).
  • Tariffs are assigned probabilistically per segment (e.g. High Value → more premium tariffs).

---

5.2 Network KPIs

fact_network_kpi_hourly (telecom_core/facts/network_kpi.py)

• Function: generate_network_kpi(dim_location, dim_time, config: NetworkConfig | None = None)
• Grain: one row per (location_id, time_id)
• Key columns:
  • location_id, time_id
  • active_users, load_ratio
  • rrc_setup_success_pct, rrc_drop_rate_pct
  • avg_throughput_dl_mbps, avg_throughput_ul_mbps
  • ux_state (Good, Degraded, Poor)
  • hasissue (bool; ux_state != "Good")
• Logic:
  • Active users driven by population_density, revenue_potential, and is_busy_hour.
  • Load ratio = active_users / capacity (capacity depends on revenue tier).
  • Throughput baseline + penalties depending on load + random noise.
  • RRC KPIs degrade under congestion + small random noise.
  • ux_state is classified from throughput and RRC KPIs.
  • hasissue is derived in one place here and reused downstream.

---

5.3 Segment Usage + Charging

fact_segment_usage_hourly (telecom_core/facts/segment_usage.py)

• Function:
  generate_segment_usage_hourly(dim_customer_segment, dim_location, dim_time, fact_network_kpi_hourly, cfg: SegmentUsageConfig, random_seed=42)
• Grain: one row per (segment_id, location_id, time_id)
• Inputs:
  • Segments: segment attributes and global mix.
  • Time: busy vs non‑busy.
  • Location: revenue tier, density (indirectly via KPIs).
  • KPIs: hasissue per location/hour.
• Outputs (per row):
  • Usage:
    • totalsessions
    • totaldatagb
    • avgsessiondurationsec
  • Charging:
    • quota_gb_hour
    • overage_data_gb
    • charge_amount_eur
    • is_throttled
    • bill_shock_flag
  • Experience:
    • affectedbyissue (network issue or throttling)
    • qualityofexperiencescore (0–1)

Logic:

1. For each hour and location, loop over all segments.
2. Simulate sessions and data using Poisson / exponential / lognormal distributions, scaled by segment and busy‑hour flags.
3. Convert monthly quotas and prices (from SegmentUsageConfig) into hourly equivalents.
4. Compute overage, charges, throttling, and bill‑shock flags.
5. Compute QoE starting from baseline_qoe_no_issue and subtracting penalties for hasissue and throttling.
6. Assemble the final fact table.

---

5.4 NOC Tickets

fact_noc_ticket (telecom_core/facts/noc_ticket.py)

• Function:
  generate_noc_tickets(fact_network_kpi_hourly, dim_time, config: NocConfig | None = None)
• Grain: one row per NOC ticket (incident spanning multiple hours)
• Key columns:
  • noc_ticket_id (e.g. NOC0010001)
  • location_id
  • start_time_id, end_time_id
  • duration_minutes
  • issue_type (e.g. Power outage, Random failure, Capacity congestion)
  • severity (Minor, Major, Critical)
• Logic:
  1. Join fact_network_kpi_hourly with dim_time to get is_busy_hour.
  2. For each location:
     • Flag hours where KPIs breach thresholds (from NocConfig) or where a random issue is drawn using NOC_BASE_ISSUE_PROB.
     • Group consecutive “bad” hours into tickets.
  3. Classify each ticket’s severity and type based on worst KPIs in the ticket window.

---

6. Pipeline Data Flow

High‑level data flow:

config/settings.py
        │
        ▼
TimeConfig ──────────► dim_time
SegmentConfig ───────► dim_customer_segment (= dim_segment)
LocationConfig ──────► dim_location
TariffConfig ────────► dim_tariff
CustomerConfig ──────► dim_customer (uses dim_location, dim_segment, dim_tariff)

(dim_location, dim_time) ──► fact_network_kpi_hourly (with hasissue)
(dim_customer_segment, dim_location, dim_time, fact_network_kpi_hourly)
    ───────────────────────► fact_segment_usage_hourly

(fact_network_kpi_hourly, dim_time)
    ───────────────────────► fact_noc_ticket

• Hard‑coded assumptions: segment profiles, location list, tariff catalog, thresholds.
• Config‑driven values: simulation dates, seeds, KPI/NOC thresholds, segment quotas/prices.
• Dynamic values: all simulated metrics (usage, KPIs, tickets) depend on random draws and config.

---

7. How to Extend

• To change the simulation period: edit SIM_START_DATE / SIM_END_DATE in config/settings.py.
• To change segment mix or behavior: update SegmentConfig in segment.py or the hard‑coded segments list.
• To change tariffs: modify the list in build_dim_tariff.
• To add a new fact table:
  1. Create a new module under telecom_core/facts/.
  2. Implement a generator function that takes the needed dimension/fact DataFrames.
  3. Wire it into telecom_core/main.py (add imports, add a new STEP with logs, write CSV).

---

If you paste this into README.md in your repo, readers should understand how to install, run, and reason about the whole pipeline.
Which part of this README would you like to go deeper into—configuration, the statistical models, or the way facts depend on dimensions?