CovidPlatformHandsOn

Welcome to the github repo for the MTC Sydney Virtual Hands On Workshop where you use Azure Data Factory, Azure Data Lake Storage, Databricks, Synapse and Power BI together to analyse the latest COVID-19 data while also building the automated data pipelines required for the analysis. The workshop has 5 modules.

This series of exercises seeks to answer the question does the number of doctors in a country affect the number of COVID-19 cases? To do this we join the latest covid-19 stats, with statistics from the GHO. You can view a whiteboard video on the overall approach here. The five modules of this workshop are outlined below.

Module 1: Ingesting external data into a data lake with Azure Data Factory Data Pipelines.

Part 1: Copying country codes data to the lake.

• You can use this Module 1 Part 1 Hands On Guide
• Download this video tutorial for part 1
• The source file for the data can be found here

Part 2: Copying the latest COVID-19 data from the John Hopkins github repo to the lake.

• You can use this Module 1 Part 2 Hands On Guide
• You can use this Module 1 Part 2 (Extension) Hands On Guide
• Download this video tutorial for Part 2 (Extention)
• The source file for the data can be found here

Part 3: Copy GHO data to the lake.

• You can use this Module 1 Part 3 Hands On Guide
• Download this video tutorial for part 3
• The service URL for the API is https://ghoapi.azureedge.net/api. You will need this when you create your OData connection.

Module 2: Wrangling the data with an open source distributed computing framework, in this case, Databricks.

Here we use in-memory techniques to wrangle the data and save a final set of files to the data lake ready to copy to the warehouse in the next module.

• For this module you'll need to import this notebook into your databricks workspace and run/modify each cell based on your work in Module 1.

Module 3: Copying the final wrangled dataset from the data lake to a Data Warehouse, here, we use Azure Synapse Analytics DW.

In this module, you create a data pipeline that copys the final dataset from Module 2 into the Azure Synapse Data Warehouse.

• You can download this video tutorial for Module 3 for your guidance.

Module 4: Creating visualizations on the warehouse data using Power BI

Finally, we create a dashboard that visualizes the data in the data warehouse.

• You can use this Module 4 Hands On Guide to do so.
• Additionally, you can also download this video tutorial for Module 4

Module 5: Creating a full data pipeline that automates all the work done above.

• You can download this video tutorial for Module 5 to help guide you.

---

author: Prerita Mehta, Viren Joseph, Microsoft Technology Center, Sydney.