Skip to content

DataPlatformModule1Part1.md

Latest commit

 

History

History
193 lines (112 loc) · 4.96 KB

DataPlatformModule1Part1.md

File metadata and controls

193 lines (112 loc) · 4.96 KB
Raw

Hands On Lab

MODULE I: DATA INGESTION

Country Code Data

MODULE 1:

Data Ingestion with Azure Data Factory

Objective 1 – Create a Pipeline to Copy the Country Codes Data Set to the Data Lake

Learning Outcomes for Module 1:

  • Importing Country Code Data from Github

  • Creating a linked service to link your Azure Storage account to the data factory. The linked service has the connection information that the Data Factory service uses at runtime to connect to it

  • Creating and validating a pipeline to copy the country code data to Azure Data Lake

Pre-requisites:

  • Azure subscription with Azure Data Factory Instance

  1. On the Let's get started page, switch to the Edit tab in the left panel.

  2. The first thing we need to do is create a folder for all our resources and work. Under the pipeline section, click the three dots for more actions and create new folder.

  1. Next, create a new pipeline. Select the plus (+) button and click on pipeline. This is going to be a data pipeline to copy stuff across

A screenshot of a cell phone Description automatically generated

  1. In the general tab, specify the pipeline name i.e. DataCopy<Name>. You will see the name of your pipeline appear on the left hand panel under Factory Resources. Drag and drop your pipeline from the left hand panel to the folder we created in the first step.

  1. Next, click on the move and transform section and drag the ‘copy data’ function and rename it to be ‘CopyCountryCodeData’ in the General panel.

  1. The Country Code data we are going to be working with in this module can be found here: https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/UID_ISO_FIPS_LookUp_Table.csv

Open this link in a new tab and click the ‘raw’ button to see this file in csv format.

  1. Back in the Azure portal, we are going to create a new source. Click on the source tab and select the (+) button to add a new dataset.

  1. On the New Dataset panel on the right hand side, select ‘HTTP’ and click continue.

  1. Given our raw data from Github is in a csv file click the DelimitedText option and click Continue.

  1. In the Set Properties panel rename the file to ‘CountryCodeData’.

  2. Select the ‘first row as header’ option.

  1. Under the Linked Service heading click ‘new’.

  2. Name the linked service to ‘HTTP<Name>Github

  3. Copy the base URL of the raw Github csv file: https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/

  4. Set Authentication type to Anonymous

  1. Click Test connection to verify that the connection is successful.

  2. Click Create.

  3. On the Set Properties Page enter Name as CountryCode

  4. Select HTTP<Name>Github as Linked service

  5. Input the relative URL of the raw Github csv file: UID_ISO_FIPS_LookUp_Table.csv

  6. Select first row as header

  7. Click OK.

  1. Select Preview Data

  1. On the Data storage read settings page select Finish.

  1. Verify that a preview of your data is displayed as shown below.

  1. Select the Sink tab and click (+) New

  1. Select Azure Data Lake Storage Gen2

  1. Select Format as Delimited Text

  1. Set Name as ‘CountryCodeDL<Name>

  2. Select first row as header

  3. Click new Linked Service

  1. Input Azure subscription as shown below

  2. Input Storage account name as shown below

  1. Click Continue

  2. On the Set Properties page select Browse under the File Path heading

  1. Select data, then inputs as shown below. Click OK.

  1. Select first row as header and click OK.

  1. Next click Validate all

  1. Once validated, click Publish all

  2. Next, click add trigger, and click trigger now

  3. Verify that the pipeline has successfully run through the storage group.

We have successfully created the Covid Country Data Pipeline. This process will be repeated twice more to create the remaining pipelines to be ingested: Covid Dataset and GHO Doctor Count.