CMPT 353 Project

Description

Table of Contents

• Installation
• Usage
  • Generating the Data
    • create-data.py
    • combine-data.py
  • Running the Analysis
    • predict-image.py
  • Generating Visualizations
    • create-map.py
    • make-graph.py

Installation

Run requirements.txt file

pip install -r requirements.txt

Usage

Generating the Data

** Note: Pre generated data is in the generated-data directory **

All of our data is created from the provided amenities-vancouver.json.gz and data
from Geoapify using their Reverse Geocoding api.

There are two scripts that generate our data:create-data.py and data.py.

create-data.py

Input

The create-data.py takes 3 command line input arguments:
argument 1: amenities-vancouver.json.gz
argument 2: amenity name from list below
argument 3: amenity name appended by _data(output dir name)

** Note: These amenities have a count from 10-37 and is recommended to run these amenities for testing. (Amenities with a high count will increase the create-data.py execution time due to the increase in api calls. Full list of amenities and their counts are in amenity_count directory.) **

List of amenity names:

ferry_terminal
trolley_bay
prep_school
college
bureau_de_change
police
bicycle_repair_station
vacuum_cleaner
clock
music_school
social_centre
compressed_air
bus_station
fire_station
marketplace
motorcycle_parking
taxi
food_court parking_space
nightclub
shower
arts_centre
bbq
events_venue
boat_rental
cinema
research_institute
university
loading_dock
weighbridge

Example command for create-data.py:
For windows:

spark-submit create-data.py amenities-vancouver.json.gz police .\generated_data\police_data

For mac:

spark-submit create-data.py amenities-vancouver.json.gz police ./generated_data/police_data

Output

Outputs a directory with given dir name and csv file in the form:
latitude | longitude | amenity | city

combine-data.py

Input

The combine-data.py takes 1 command line input arguments:
argument 1: amenities-vancouver.json.gz

Example command for combine-data.py:

spark-submit combine-data.py amenities-vancouver.json.gz

Output

Outputs amenities-vancouver.csv in the form:
latitude | longitude | city | count | amenity

---

Running the Analysis

There is 1 script that generate our analysis: predict-image.py.

predict-image.py

The predict-image.py file takes in an image as input and predicts where the image was taken.

Input

The predict-image.py takes 2 command line input arguments:
argument 1: .jpg file
argument 2: amenities-vancouver.csv

** Note: sample input images can be found in the sample-images directory **

Example command for predict-image.py:
For windows:

spark-submit predict-image.py .\sample-images\IMG_2991.jpg amenities-vancouver.csv

For mac:

spark-submit predict-image.py ./sample-images/IMG_2991.jpg amenities-vancouver.csv

Output

Outputs the following to the console:
time iamge was taken, lat/lon, training score, validation score, predicted city

---

Generating Visualizations

** Note: Pre generated visualizations can be found in the visualizations directory **

There are 2 scripts that generate visuals: create-map.py and make-graph.py

create-map.py

The create-map.py plots points on a map giving a visual of the distribution of the amenities.

Input

The create-map.py takes 1 command line input argument:
argument 1: csv from generated-data

Example command for create-map.py:
For windows:

python create-map.py .\generated-data\bank_data\part-00000-0133636f-dfa8-40a0-9c43-4ce07c07c1bd-c000.csv

For mac:

python create-map.py ./generated-data/bank_data/part-00000-0133636f-dfa8-40a0-9c43-4ce07c07c1bd-c000.csv

Output

Outputs a heat_map.html and marker_map.html file which can be opened with a browser to see the map.

make-graph.py

The make-graph.py creates a scatter plot of all the generated data points created in the generated-data directory.

Input

The make-graph.py takes 1 command input argument:
argument 1: amenities-vancouver.csv

Example command for make-graph.py:

spark-submit make-graph.py amenities-vancouver.csv

Output

Outputs map_of_amenities.png which is scatter plot of all amenities in each color coded city. Outputs *.png which is a scatter plot of all the amenities in amenities-vancouver.csv .