Network Log Analytics AI

A real-time cybersecurity analytics platform for detecting lateral movement attacks from enterprise authentication and network logs using machine learning.

The project combines synthetic enterprise event generation, Apache Kafka-based event streaming, behavioural feature engineering, anomaly detection using Isolation Forest, MongoDB storage, a FastAPI backend, and a React-based monitoring dashboard to simulate an enterprise Security Operations Center (SOC) workflow.

---

Overview

Lateral movement is one of the most critical stages of a cyber attack, where an attacker who has gained initial access attempts to move across internal systems to escalate privileges or access high-value assets.

Network Log Analytics AI analyzes authentication and internal network events to identify abnormal behaviour patterns associated with:

• Lateral Movement
• Privilege Escalation
• Internal Reconnaissance
• Suspicious Authentication Behaviour
• Abnormal Host Traversal
• Administrative Protocol Misuse

Rather than relying solely on rule-based detection, the system engineers behavioural features from authentication logs and applies an Isolation Forest model to identify anomalous activity.

---

Key Features

• Synthetic enterprise authentication and network event generation
• Real-time event streaming using Apache Kafka
• Behavioural feature engineering for lateral movement detection
• Isolation Forest based anomaly detection
• MongoDB event storage
• FastAPI REST API
• Interactive React monitoring dashboard
• Offline model training and evaluation pipeline
• Dockerized infrastructure

---

What It Detects

The pipeline models lateral movement as abnormal internal user-host traversal after an attacker has already entered the network.

Detection is based on behavioural indicators such as:

• One user accessing many new destination hosts within a short time window
• Unusual source-to-destination host relationships
• Administrative protocol usage (RDP, WinRM, SMB, SSH)
• Successful authentication after repeated failures
• Privileged account activity outside normal behaviour
• Rapid lateral hops across workstations, servers and domain controllers
• Rare protocol usage
• Abnormal bytes-out ratios
• Off-hours authentication activity

---

Technology Stack

Backend

• Python
• FastAPI
• Apache Kafka
• MongoDB
• Docker

Machine Learning

• Scikit-learn
• Isolation Forest
• Pandas
• NumPy
• Joblib

Frontend

• React
• TypeScript
• TanStack Router
• Recharts

---

Current Model Performance

The model was evaluated on a held-out test dataset containing 9,885 authentication events that were never used during training or threshold calibration.

Metric	Score
Accuracy	98.30%
Precision	89.00%
Recall	97.98%
F1-score	93.27%
ROC-AUC	0.9967
PR-AUC	0.9876

Confusion Matrix

	Predicted Normal	Predicted Lateral Movement
Actual Normal	8552	144
Actual Attack	24	1165

Decision Threshold

0.623875

The model is trained as an unsupervised anomaly detector where Isolation Forest learns normal authentication behaviour before a calibrated threshold is selected using a labeled validation dataset.

---

Main Pipeline

1. producer.event_builder generates synthetic enterprise authentication and network events.
2. scripts.generate_synthetic_data creates labeled CSV and JSONL datasets.
3. ml_engine.feature_builder computes behavioural features used for anomaly detection.
4. ml_engine.train trains the Isolation Forest model and saves model artifacts.
5. ml_engine.evaluate evaluates the trained model and exports metrics to outputs/metrics.json.
6. producer.producer continuously streams live authentication events into Apache Kafka.
7. consumer.consumer performs feature engineering, anomaly scoring and stores enriched events in MongoDB.
8. api.main exposes analytics through FastAPI.
9. The React dashboard visualizes live events, threat analytics, machine learning metrics and system health.

---

Behavioural Features

The feature engineering pipeline extracts behavioural indicators including:

• unique_dst_hosts_5m
• unique_src_hosts_5m
• unique_protocols_5m
• auth_event_rate_5m
• failed_auth_count_5m
• success_after_fail_count_10m
• new_user_host_pair
• new_src_dst_pair
• rare_protocol_flag
• privileged_account_flag
• admin_protocol_flag
• lateral_hop_count_15m
• fanout_score_15m
• bytes_out_ratio
• off_hours_flag

---

Project Structure

NetworkLogAnalyticsAI/

├── api/
├── configs/
├── consumer/
├── datasets/
├── docker/
├── Frontend/
├── ml_engine/
├── producer/
├── scripts/
├── shared/
├── outputs/
├── docker-compose.yml
├── requirements.txt
└── README.md

---

Reproduce the Model

Generate the synthetic dataset:

python -m scripts.generate_synthetic_data --events 30000 --seed 42

Build behavioural features:

python -m ml_engine.feature_builder

Train the model:

python -m ml_engine.train

Evaluate the model:

python -m ml_engine.evaluate

Generated artifacts:

• datasets/synthetic/network_auth_events.csv
• datasets/synthetic/network_auth_events.jsonl
• datasets/processed/training_dataset.csv
• ml_engine/model.pkl
• ml_engine/scaler.pkl
• ml_engine/threshold.json
• outputs/metrics.json

---

Running the Streaming Pipeline

Start the infrastructure:

docker compose up -d

Run the producer:

python -m producer.producer

Run the consumer:

python -m consumer.consumer

The consumer automatically loads:

• ml_engine/model.pkl
• ml_engine/scaler.pkl
• ml_engine/threshold.json

If model artifacts are unavailable, events are still collected and stored for future model training.

---

Running the Dashboard

Start the backend:

uvicorn api.main:app --reload --port 8000

Start the frontend:

cd Frontend/Dashboard
npm install
npm run dev

---

REST API

Endpoint	Description
GET /api/summary	Dashboard summary statistics
GET /api/events	Recent network authentication events
GET /api/ml	Machine learning metrics
GET /api/system	System status and health information

---

Dashboard Modules

The React dashboard currently includes:

• Overview Dashboard
• Live Event Monitor
• Threat Analysis
• Machine Learning Analytics
• User Behaviour
• System Status

The dashboard communicates directly with the FastAPI backend and displays live information from MongoDB.

---

Notes for Review

This project supports both labeled and unlabeled workflows.

• Synthetic labeled datasets are used for reproducible training and quantitative evaluation.
• Isolation Forest performs anomaly detection without requiring labels during inference.
• Authentication logs from enterprise environments can be adapted to the same event schema for future deployment.

---

Future Enhancements

• Windows Event Log integration
• Sysmon support
• SIEM integration
• Explainable AI for anomaly interpretation
• Threat intelligence enrichment
• Alerting and notification pipeline
• Real enterprise authentication datasets

---

Author

• Rahul Karmakar ( Aspiring Data Analyst | Astrophysicist)
• Associated with CDAC Chennai