Skip to content

DataPlatformModulePart2.md

Latest commit

 

History

History
169 lines (97 loc) · 4.41 KB

DataPlatformModulePart2.md

File metadata and controls

169 lines (97 loc) · 4.41 KB
Raw

Hands On Lab

MODULE I: DATA INGESTION

COVID-19 Dataset

MODULE 1:

Data Ingestion with Azure Data Factory

Objective 2 – Create a Pipeline to Copy Daily COVID-19 Data with Parameters to the Data Lake

Pre-requisites:

  • Azure subscription with Azure Data Factory Instance

  • Completed Objective 1 of Module 1: Creating a pipeline to copy country code dataset to the Azure Data Lake

  1. Create a new pipeline through selecting the plus (+) button and click on pipeline

A screenshot of a cell phone Description automatically generated

  1. In the general tab, specify the pipeline name i.e. Covid Latest Pipe. You will see the name of your pipeline appear on the left hand panel under Factory Resources.

  1. Next, click on the move and transform section and drag the ‘copy data’ function and rename it to be ‘CovidDataCopy’ in the General panel.

  1. The dataset we will be working with for this module can be found here: https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports

  1. Next up, we are going to create a new source. Click on the source tab and select the (+) button to add a new dataset.

  1. On the New Dataset panel on the right hand side, select ‘HTTP’ and click continue.

  1. Given our raw data from Github is in a csv file click the DelimitedText option and click Continue.

  1. In the Set Properties panel rename the file to ‘CovidDataSource’.

  1. Under the Linked Service heading click ‘new’.

  2. Name the linked service to ‘HTTPServerReferenceLinkedService’

  3. Copy the base URL of the raw Github csv file as shown below (https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/)

  4. Set authentication to Anonymous

  1. Click Test connection to verify that the connection is successful.

  1. Click Create.

  2. Click OK.

  3. Click Open Source dataset

  1. To ensure we pull the data with the latest date we must set a parameter. Navigate to the Parameters tab and click new.

  2. Input ‘FileDate’ in the name field.

  1. Navigate to the Connection tab. Under the Relative URL field click on add dynamic content, as we want to ensure we always pull the latest COVID-19 data.

  1. What we want to do here is get the FileDate as a csv. Input the following in the add dynamic content field:

  2. Click Finish

  3. Return back to your actual pipeline to test that our parameters work. Input a file date to test, and click on Preview data.

  1. Next up, we have to define the data lake that we want to copy this data to. Navigate to the Sink tab and click new.

  2. Select Azure Data Lake Storage Gen 2

  1. Select Delimited Text

  1. Input ‘CovidLatest’ in the name field

  1. Select new linked service

  2. Input fields as shown below:

Name: ADLSLSReference

Azure subscription: MTC Sydney Azure

Storage account name: stcovidhackoutputprod

  1. Click test connection.

  1. Next, to set a file path click browse

  1. Click on data and then select inputs

  1. Click OK.

  2. Click Validate All

  3. If all the validations are correct, click Publish All

  4. Next, click add trigger

  1. Click Trigger now to trigger the pipeline straight away.

  2. To check how the pipeline is running, go to the monitor step in Azure Data Factory.