This project implements a sliding aggregation algorithm for processing large datasets, with a specific focus on NYC Taxi data. The project leverages the power of Spark and Hadoop for efficient data processing and analysis. The primary goal is to perform sliding aggregation on the dataset and evaluate the results.
Before running the code, make sure you have the following prerequisites:
- Python
- Jupyter Notebook
- Spark
- Hadoop
- PyDrive
- OpenJDK 8
The code is organized into several sections:
-
Configuring Spark: This section installs and configures the necessary dependencies, including Spark, PyDrive, and OpenJDK.
-
Creating Spark Session: It creates a Spark Session for data processing, which includes setting memory configurations.
-
Download Data: The script downloads the NYC Taxi data in Parquet format for analysis.
-
Prepare Data for the Assignment: This section prepares the data by selecting specific columns and adding queries and label placeholders.
-
TeraSort: The TeraSort algorithm is used to sort and balance the data efficiently.
-
Perfect Balance: The script calculates the rank using prefix sums to achieve perfect balance in the dataset.
-
Multidimensional Interval Multiquery Processor: This section processes multidimensional interval queries on the balanced dataset.
-
All in One Place: A function is provided to run the entire processing pipeline, including sorting, balancing, and query processing.
To use this code, follow these steps:
-
Install the required dependencies mentioned in the "Prerequisites" section.
-
Download the NYC Taxi data or replace it with your own dataset.
-
Open the Jupyter Notebook and run each code cell sequentially.
-
Modify the code as needed for your specific dataset and analysis.
The code includes ideas and concepts from the following research papers: