Data migration to Yelo SaaS platform

This repo was build to achive the next goals:

• Clean the client data
• Upload that data to the Yelo SaaS platform

Challanges

• At the moment of the creation of this repo, the Yelo SaaS platform does not support the bulk upload of data via API.
• Upload the data as a CSV file is not an option because the data is too big (2M records) and the CSV doesn't support all the fields.
• There're only individual endpoints, so upload the data customer by customer takes too much time.

Solution

• Use an async and concurrent algorithm for speed
• Use a semaphore to avoid reaching the limit of the API (300 hits per sec)

Setup

• Only need to create the env vars for input/output files

Environment variables

Name	Description
YELO_API_KEY	API key
YELO_API_BASE_URL	Base URL
RAW_DATA_DIR	Directory where the raw data is stored
RAW_DATA_FILE_NAME	Name of the file with the raw data
CLEAN_DATA_DIR	Directory where the clean data is stored
CLEAN_DATA_FILE_NAME	Name of the file with the clean data
RESULTS_DIR	Directory where the results of the upload attempt are stored
RESULTS_FILE_NAME	Name of the file with the results of the upload attempt