Cloud Data Engineer — Project Repository

Welcome to the Cloud Data Engineer lab — a portfolio-style repository that showcases real-world data engineering and data science projects using modern cloud technologies such as AWS, GCP, and open public data.

This repository is structured to reflect professional development practices, with each subproject demonstrating specific cloud pipelines, automation techniques, and scalable architectures.

---

Subprojects

ETL: ANEEL Complaint Data to AWS

Extracts consumer complaint data from the ANEEL public API, transforms the dataset using Pandas, and uploads it to an AWS S3 bucket.

• Technologies: Python, AWS S3, Boto3, Poetry, Taskipy
• Focus: REST API ingestion, transformation, cloud-native ETL pipeline
• Automation: Run and deploy using task run and task deploy

---

Hadoop Ecosystem — PySpark Word Count

Runs a distributed word count job using PySpark on Google Cloud Dataproc, processing a text file stored on GCS and saving the sorted word frequencies.

• Technologies: PySpark, Google Cloud Dataproc, GCS
• Focus: Big Data processing with Hadoop ecosystem
• Execution: gcloud dataproc jobs submit pyspark ...

---

Tools & Technologies

• Cloud Platforms: AWS (S3), GCP (Dataproc, GCS)
• Languages & Frameworks: Python, PySpark
• Package & Task Management: Poetry, Taskipy
• Libraries: Pandas, Requests, Boto3

---

How to Use

Clone the repository:

git clone https://github.com/leticiagcsilva/Cloud_Data_Engineer.git
cd Cloud_Data_Engineer