SPAD-based Nitrogen Level Assessment using Data Augmentation and EDA

This repository contains the code and resources for the project "Data Minors - 9426 ITE 17". The project focuses on exploring, preprocessing, and augmenting SPAD (Soil Plant Analysis Development) data and corresponding plant images to assess nitrogen levels.

🚀 Quick Links

• Project Repository: https://github.com/ReyJrFerrer/Data-Mining-BSU.git
• Live Dashboard: https://spad-dashboard.streamlit.app/

👥 Team Members

• Dela Concha, Don Daryll
• Dela Cruz, Perry
• Ferrer, Reynaldo Jr.
• Flordeliz, Ron Arvene
• Liquigan, Ralph Aloven
• Lucina, Christian
• Zarate, Jan Dale

📝 Project Overview

The project is divided into four main phases, starting from initial data exploration and preprocessing, moving to image manipulation, and finally data and image augmentation to create a more robust dataset for model training.

---

📂 Project Phases

1. SPAD Data Exploration and Preprocessing

This initial phase focuses on understanding and cleaning the raw SPAD data.

• data-preprocessing.ipynb: A Jupyter notebook that covers the initial preprocessing steps and a simple Exploratory Data Analysis (EDA) to understand the data distributions and relationships.
• EDA Dashboard: A comprehensive EDA is available on the interactive dashboard, which provides visualizations and insights into the dataset and images representing spad value ranges. This can be viewed at spad-dashboard.streamlit.app.

2. Image Preprocessing

This phase deals with cleaning and preparing the plant images for further processing.

• Background Removal: The background-lab-remover.py script is used to programmatically remove the background from images taken in a laboratory setting.
• Manual Cropping: After background removal, images are manually cropped to remove the paper containing the SPAD value, ensuring that only the plant is visible.
• Output: The cleaned and preprocessed images are stored in the SPAD-Processed folder.

3. Data Augmentation

This phase focuses on synthetically increasing the size of the tabular dataset to improve model performance and generalization.

• data-augmentation.ipynb: This notebook contains the script for data augmentation techniques applied to the SPAD dataset.
• Output: The script generates augmented_potato_combined.csv. This file will be renamed to augmented_cabbage_potato_combined.csv and will be used as an input for the image augmentation phase.

4. Image Augmentation

In this final phase, we generate synthetic images based on the augmented data to create a larger and more diverse image dataset.

• field-image-transfer.py: This script is used to transfer field images to the SPAD-Processed directory to be used in the augmentation process.
• image-augmentation.py: The core script for performing image augmentation. It generates new images based on the transformations and data from the augmented CSV.
• image-augmentation-eda.py: This script is used to perform an EDA on the newly generated augmented images to evaluate their quality and distribution.

---