🏙️ Cost of Living Analytics & Household Segmentation

📌 Executive Summary

Understanding regional affordability and the true drivers of the cost of living is a critical challenge for economic planners, policymakers, and businesses. This project analyzes a massive dataset of over 31,000 US counties and households to uncover the underlying patterns of regional expenses. By building a rigorous data wrangling pipeline, deploying predictive regression models, and utilizing unsupervised machine learning (clustering), this analysis segments the US population into distinct financial personas and accurately models the primary drivers of household financial burden.

💡 Business Q&A & Actionable Insights

Q1: What are the primary drivers of total household costs?

• A: Direct expense inputs—specifically housing, food, and transportation—overwhelmingly dominate total household costs. Geographic location alone adds surprisingly little predictive power once these core expenses are accounted for.
• Action: Affordability policies and cost-prediction models should prioritize stabilizing core necessities (like housing and food) over generalized regional subsidies.

Q2: Are there distinct "financial personas" among US households?

• A: Yes. K-Means clustering (validated by Silhouette Scores) identified six distinct household segments. For example, Cluster 5 represents affluent, metro-based families with the highest median income and extreme childcare costs (averaging ~$12,900). Conversely, Clusters 3 and 4 represent smaller, non-metro households (avg. 1.9 members) with minimal childcare and significantly lower housing costs.
• Action: Businesses offering family services or regional planners can use these 6 personas to tailor targeted interventions (e.g., childcare subsidies heavily targeted at metro families).

Q3: How much do state lines and geography impact specific living costs?

• A: While total costs are driven by core expenses, specific categories like taxes and healthcare exhibit massive geographic variance. Taxes, for example, have a standard deviation of nearly $2,500, highlighting the severe impact of state-level tax policies.
• Action: Relocation planning and regional economic forecasting must explicitly model local tax and healthcare structures rather than relying on national averages.

Q4: Can we accurately predict the total cost of living using sub-expense data and demographics?

• A: Yes. Using a Random Forest Regressor, the model achieved high R² values and low error rates, successfully capturing the non-linear interactions between family size, location type (metro vs. rural), and baseline expenses.
• Action: Institutions can deploy this regression pipeline to reliably estimate cost-of-living adjustments for employees or citizens.

🧹 Data Wrangling & Exploratory Data Analysis (EDA)

Real-world economic data requires extensive preparation to prevent skewed modeling. A strict preprocessing pipeline was developed:

• Feature Engineering: Extracted raw household string codes (e.g., "1p2c") using Regular Expressions into robust numerical features (num_parents, num_children, family_total), ultimately dropping redundant features to eliminate multicollinearity.
• Outlier Capping (Winsorization): Applied the IQR method and winsorize from SciPy to strictly cap extreme local variations at the 5th and 95th percentiles for continuous variables.
• Skewness Normalization: Executed natural log transformations (np.log1p) on highly right-skewed features (housing costs, taxes, and median family income) to stabilize variance.
• Geographic Consolidation: Mapped state-level data to the four major US Census Regions (Northeast, Midwest, South, West) to reduce categorical cardinality for clustering.

📈 Predictive Regression Modeling

To predict total household costs, the data was partitioned (70% Train, 20% Val, 10% Test) and fed into regression pipelines:

• Lasso Regression: Provided a strong linear baseline and performed feature selection via L1-regularization.
• Random Forest Regressor: The champion model. It significantly improved predictive accuracy by capturing the complex, non-linear relationships (such as the interaction between family size and childcare costs) that linear models could not.

🤖 Unsupervised Machine Learning (Clustering)

To segment the population into actionable profiles, unsupervised clustering was applied to scaled (Min-Max) and encoded data:

• K-Means Clustering: Evaluated K=2 through K=10 using Inertia and Silhouette Scores. K=6 emerged as the optimal structure (Silhouette Score: 0.299), successfully grouping households based on spending behaviors.
• Hierarchical Clustering (Agglomerative): Built a dendrogram to validate the nested structure of the data, providing a complementary, exploratory view of consumer behavior.
• Dimensionality Reduction: Applied Principal Component Analysis (PCA) to compress the 14-feature dataset into a 2D space, allowing for clear visual validation of the 6 household clusters.

🛠️ Tech Stack

• Language: Python
• Data Wrangling: Pandas, NumPy, Regular Expressions (re)
• Data Visualization: Matplotlib, Seaborn
• Statistical Analysis: SciPy (Winsorization, Chi-Square Testing)
• Machine Learning: Scikit-learn (Lasso, Random Forest, K-Means, AgglomerativeClustering, PCA, GridSearchCV)

📁 Repository Structure

├── data/
│   └── cost_of_living_us.csv         # Raw dataset 
├── notebooks/
│   ├── 01_Cleaning.ipynb             # Outlier handling and feature engineering
│   ├── 02_EDA.ipynb                  # Distributions and correlation analysis
│   ├── 03_Preprocessing.ipynb        # Log transformations and scaling pipelines
│   ├── 04_Regression_Models.ipynb    # Lasso and Random Forest predictive models
│   └── 05_Cluster_Models.ipynb       # K-Means, Hierarchical clustering, and PCA
├── README.md                         # Project overview and insights

---