Los Angeles Crime Data Analysis

A big data processing system for analyzing Los Angeles crime data, implementing a data lake architecture with both batch and stream processing pipelines. This project is made as part of a college course.

Datasets

Primary Datasets (Historical Crime Data)

• Historical Dataset I (2010-2019): Crime Data from 2010 to 2019

  • File: crime_data.csv

• Historical Dataset II (2020-Present): Crime Data from 2020 to Present

  • File: crime_data_2020_to_present.csv

Secondary Dataset (Stream Data)

• Real-time 911 Calls: Simulated emergency call stream
  • Generated by: Kafka/producer.py
  • Purpose: Simulating real-time stream processing and operational analytics

Architecture

The system implements a complete data lake architecture with three logical stacks:

1. Batch Processing Stack: MongoDB → Spark → Airflow orchestrates PySpark jobs
2. Stream Processing Stack: Kafka producer → Java Kafka Streams processor → MongoDB
3. Analytics Stack: Metabase for visualization and dashboards

Analytical Questions

Batch Processing

1. What area is the most risky?
2. In which area were the most cases resolved?
3. What is the risk distribution for areas?
4. What are the reporting speeds of cases?
5. How long did each area take to respond on average?
6. What is the relationship between risk and crime density?
7. What year had the most crimes committed?
8. How is risk changing over the years?
9. What are the latest trends for crime increase/decrease for areas?
10. Year over Year change of crime for each area?
11. Which areas are most dangerous for female victims?
12. What is the vulnerability score for different age groups and genders by area?
13. What are the safest and riskiest times (hour/day) for women in each area?
14. What is the overall crime distribution for women?
15. What is the crime breakdown for women for each area?
16. What is the crime count per hour of each day for weekdays and weekends?

Stream Processing

1. Are there calls that the priority should be escalated for?
2. Is the response time degrading in some areas?
3. Is there a spike in calls (anomaly detection)?
4. What is the utilization of emergency units (LAPD/LAFD/EMS)?
5. What is the dispatch failure rate in high-risk areas?

Setup Guide

Prerequisites

• AWS credentials configured (~/.aws/credentials with private_account profile)
• Historical datasets: crime_data.csv and crime_data_2020_to_present.csv in ./Airflow/files/

Starting the Cluster

# Start entire cluster (all stacks)
./scripts/cluster_up.sh

# Or start individual stacks
./scripts/batch_up.sh      # MongoDB + Spark + Airflow
./scripts/stream_up.sh     # Kafka + Stream Processor
./scripts/analytics_up.sh  # Metabase

Shutting Down

# Shutdown entire cluster
./scripts/cluster_down.sh

# Or shutdown individual stacks
./scripts/batch_down.sh
./scripts/stream_down.sh
./scripts/analytics_down.sh

Data Specifications

• Crime Codes: UCR Crime Classification
• Weapons Used: Weapon Categories
• Project Requirements: Technical Specification

Analytics Dashboards

Batch Processing Reports

• Business Metrics Report - Resolution rates, response times, risk scores, and YoY trends
• Demographic Safety Report - Female vulnerability analysis, age-gender vulnerability scores, and temporal safety patterns