Skip to content

DataPlatformModule1Part2Easy.md

Latest commit

 

History

History
165 lines (95 loc) · 4.25 KB

DataPlatformModule1Part2Easy.md

File metadata and controls

165 lines (95 loc) · 4.25 KB
Raw

Hands On Lab

MODULE I: DATA INGESTION

COVID-19 Dataset

MODULE 1:

Data Ingestion with Azure Data Factory

Objective 2 – Create a Pipeline to Copy Daily COVID-19 Data with Parameters to the Data Lake

Pre-requisites:

  • Azure subscription with Azure Data Factory Instance

  • Completed Objective 1 of Module 1: Creating a pipeline to copy country code dataset to the Azure Data Lake

Learning Outcomes for Module 1:

  • Importing COVID-19 Data from Github

  • Creating a linked service to link your Azure Storage account to the data factory. The linked service has the connection information that the Data Factory service uses at runtime to connect to it

  • Creating and validating a pipeline with specified parameters to copy the COVID-19 data to Azure Data Lake

  1. Create a new pipeline through selecting the plus (+) button and click on Pipeline

A screenshot of a cell phone Description automatically generated

  1. In the general tab, specify the pipeline name i.e. Covid Latest Pipe. You will see the name of your pipeline appear on the left hand panel under Factory Resources.

  1. Next, click on the move and transform section and drag the ‘copy data’ function and rename it to be ‘CovidDataCopy’ in the General panel.

  1. The dataset we will be working with for this module can be found here: https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports

  1. Next up, we are going to create a new source. Click on the source tab and select the (+) button to add a new dataset.

  1. On the New Dataset panel on the right hand side, select ‘HTTP’ and click continue.

  1. Given our raw data from Github is in a csv file click the DelimitedText option and click Continue.

  1. In the Set Properties panel rename the file to ‘CovidDataSource’.

  1. Under the Linked Service heading click ‘new’.

  2. Name the linked service to ‘HTTPServerReferenceLinkedService’

  3. Copy the base URL of the raw Github csv file as shown below https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/

  4. Set authentication to Anonymous

  1. Click Test connection to verify that the connection is successful.

  1. Click Create.

  2. Input the date of the particular raw csv field in the Relative URL field e.g. 05-16-2020.csv

  3. Click OK.

  4. Next, click on Preview data as shown

  1. Next up, we have to define the data lake that we want to copy this data to. Navigate to the Sink tab and click new.

  2. Select Azure Data Lake Storage Gen 2

  1. Select Delimited Text

  1. Input ‘CovidLatest’ in the name field

  1. Select new linked service

  2. Input fields as shown below:

Name: ADLSLSReference

Azure subscription: MTC Sydney Azure

Storage account name: stcovidhackoutputprod

  1. Click test connection.

  1. Next, to set a file path click Browse

  1. Click on data and then select inputs

  1. Click OK.

  2. Click Validate All

  3. If all the validations are correct, click Publish All

  4. Next, click Add trigger

  1. Click Trigger now to trigger the pipeline straight away. When prompted for a parameter, input a date value in the format of MM-DD-YYYY. Click OK. The pipeline will then start to run.

  2. To check how the pipeline is running, go to the monitor step in Azure Data Factory.