Data Engineering group project for UT class LTAT.02.007.
Before starting Airflow for the first time, You need to prepare your environment, i.e. create the necessary files, directories and initialize the database.
On Linux, the quick-start needs to know your host user id and needs to have group id set to 0. Otherwise the files
created in dags, logs and plugins will be created with root user. You have to make sure to configure them for the
docker-compose:
Run mkdir -p ./dags ./logs ./plugins
Run echo -e "AIRFLOW_UID=$(id -u)" > .env
Running Airflow services (add -d to run in background):
docker-compose up
Once the cluster has started up, you can log in to the web interface and try to run some tasks.
- The Airflow webserver is available at:
http://localhost:8080. The default account has the loginairflowand the passwordairflow. - The Neo4J Browser UI is available at:
http://localhost:7474with usernameneo4jand passwordchangeme.
The following command stops containers and removes containers, networks, volumes, and images created
by docker-compose up.
docker-compose down --volumes --remove-orphans
Data generated by the Airflow pipeline is imported into following two databases.
MongoDB upload step is dependant on the configuration specified in dags/mongo_upload.py.
In order to run the queries, specify MONGO_SECRET value in queries/mongodb_queries.py#4.
Then run the queries from IDE or from commandline using the python executable, e.g.
python3 queries/query.py
Example queries that can be run (e.g. via Neo4J Browser UI) after neo4j_upload Airflow task has finished importing data & creating relations:
- Finding the longest path of parent-child relations in memes
MATCH (child)-[r:CHILD]->(parent) RETURN parent, COLLECT(child) as childs ORDER BY SIZE(childs) desc limit 1
- Finding the meme with most child relations
MATCH p=(start:Meme)-[:CHILD*1..10]->(end:Meme) RETURN p ORDER BY length(p) DESC LIMIT 1