Yelp Data Processing and Analysis with Hadoop and Spark

This project focuses on processing and analyzing Yelp data using Hadoop and Spark. The dataset includes business reviews, user data, and check-ins from Yelp. The project is split into multiple parts: data storage, data processing, data analysis, and data visualization.

---

Overview

In this project, we process the Yelp dataset using Hadoop, performing various tasks like data cleaning, deduplication, filtering, and analysis. The data is then analyzed using Spark to generate insights about business performance, customer engagement, and ratings. The final step involves visualizing the results using Python's Matplotlib and Seaborn libraries.

---

Steps Overview

Part 1: Dataset

• Summary: Download the Yelp dataset, which contains multiple JSON files (business, review, user, checkin, and tip). A PDF file is provided to show the structure of these JSON files.

Part 2: Data Storage

• Summary: Upload the dataset to HDFS (Hadoop Distributed File System) and organize it into directories. Verify the successful upload of all files. This step sets up the data for processing using Hadoop.

Part 3: Data Processing with Hadoop

• Summary:
  • Process business.json and review.json.
  • Filter out invalid businesses and deduplicate data.
  • Extract relevant business data (ID, name, city, categories, rating).
  • Count reviews and calculate engagement metrics (useful, funny, cool votes).
  • Combine business and review data into a final dataset.

Part 4: Data Analysis with Hadoop or Spark

• Summary:
  • Perform analysis using either Hadoop or Spark.

Part 5: Data Visualization

• Summary:
  • Visualize the analysis results using Python’s Matplotlib and Seaborn libraries.
  • Create charts and graphs to better understand business performance and customer engagement trends.

---