🚀 Signal Extraction ML Pipeline

A Snowflake-powered end-to-end machine learning pipeline for financial signal generation and prediction.

---

---

📖 Overview

🏆 Hackathon Runner-Up in Snowflake's Dev Premier League!

This project was developed as part of the Snowflake Hackathon, and secured a Runner‑Up position by building a complete data ingestion → transformation → prediction → visualization pipeline.

It leverages Snowflake’s Data Cloud, Snowpark, and Python ML libraries to extract meaningful trading signals from financial and news data.

The goal: generate predictive buy/sell signals by combining market price movements and news sentiment analysis — all within a scalable Snowflake architecture.

---

🏆 Achievements

• Runner-Up: Secured Runner-Up place in Snowflake's Dev Premier League Hackathon for innovative ML pipeline design.
• End-to-End Solution: Fully automated pipeline from data ingestion to visualization, deployed on Snowflake.
• Real-World Impact: Demonstrated predictive trading signals with explainability and backtesting.

---

✨ Key Features

Feature	Description
📊 Data Ingestion	Automated collection of stock prices via Alpha Vantage API and financial news via News API
🔧 Feature Engineering	Technical indicators (EMA, RSI, MACD, Volatility) + sentiment features (polarity, subjectivity, entity signals, recency)
🔍 Explainability	SHAP-based model explanations & feature importance for predictions and sentiment analysis
📈 Backtesting	(Under Development) Event-driven engine with PnL, Sharpe ratio, and drawdown analysis
⚙️ Orchestration	Snowflake Tasks & Streams (Airflow-compatible) for end-to-end pipeline automation (ingestion → transform → training → scoring → backtests)
🤖 ML Pipeline	XGBoost + Gradient Regressor with Snowpark; AI_CLASSIFY for sentiment labeling and an annotated dataset maintained for live news to improve and validate classifiers.
📱 Interactive Dashboard	Streamlit app with signal explorer, explainability charts, and performance metrics

---

🧰 Tech Stack

Layer	Technology	Purpose
Data Storage	Snowflake (SIGNAL_EXTRACTION_DB)	Centralized data warehouse
Compute	Snowflake Warehouse (COMPUTE_WH)	Scalable compute for ETL + ML
Ingestion	Python, REST APIs	Pulls stock + news data
Transformation	Snowflake SQL, Snowpark	Data cleaning and feature creation
ML	Python (XGBoost, Pandas), Snowflake Libs	Model training & prediction
Visualization	Streamlit	Application & Reporting layer

---

🧩 High Level Data Pipeline (Architecture)

🗂️ Use Case & Repository Structure

🧑‍💼 Use Case Diagram (click to expand)

---

🗂️ Repository Structure (click to expand)

📦 signal-extraction-ml-pipeline
├── 📁 src/
│   ├── 1_ingestion/
│   │   ├── 1_ingest_market_api.ipynb
│   │   ├── 1_ingest_news_api.ipynb
│   │   ├── market_config.json
│   │   └── news_config.json
│   ├── 2_transformation_and_feature_engineering/
│   │   └── 1_transformation_and_feature_engineering_market_data.sql
│   ├── 3_ml/
│   │   ├── 1_analyze_news_data.ipynb
│   │   ├── 1_predict_market_data.ipynb
│   │   ├── environment.yml
│   │   └── market_config.json
│   ├── 4_frontend/
│   │   ├── streamlit_app.py
│   │   ├── environment.yml
│   │   └── market_config.json
│   ├── 📁 infra/
│   ├── 📁 docs/
│   ├── 📁 env/  # Virtual environment (auto-generated)
│   └── 📁 utils/
├── [requirements.txt]
├── [README.md]
└── LICENSE

---

📈 Visualization & Outputs

📈 Visualizations (click to expand)

📈 Price Prediction Graph

🔍 Price Explainability

📊 Model Performance Summary

📰 News Sentiment Ranked

---

📊 Underlying Data Outputs (click to expand)

📋 Underlying Chart Data

📦 Indicators JSON Dump

---

⚙️ Setup & Installation

⚙️ Steps (click to expand)

1️⃣ Prerequisites

• Python 3.9+
• Snowflake account with appropriate roles
• Alpha Vantage & News API keys
• Snowflake Python Connector installed

2️⃣ Clone the Repository

git clone https://github.com/<your-username>/signal-extraction-ml-pipeline.git
cd signal-extraction-ml-pipeline

3️⃣ Install Dependencies

pip install -r requirements.txt

4️⃣ Configure Environment

Edit market_config.json & news_config.json:

{
  "...": "...",
  "snowflake": {
    "account": "XXXXXX",
    "user": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD",
    "role": "ACCOUNTADMIN",
    "warehouse": "COMPUTE_WH",
    "database": "SIGNAL_EXTRACTION_DB",
    "schema": "<CODE>"
  }
}

5️⃣ Run Pipeline

# Infra Setup
snowsql infra/{code}
# Ingestion Execution
python src/1_ingestion/1_ingest_market_api.ipynb
python src/1_ingestion/1_ingest_news_api.py
# Transformation & Feature Generation Execution
execute 2_transformation_and_feature_engineering/1_transform_and_feature_engineering_market_data.sql
python 2_transformation_and_feature_engineering/1_generate_news_articles_features.ipynb
# ML Execution
python src/3_ml/1_analyze_news_data.sql
python src/3_ml/1_predict_market_data.ipynb
python src/3_ml/2_backtest_market_data.ipynb
# Visualization Application Setup
python src/4_frontend/streamlit_app.py

---

🎬 Submission & Showcase Resources

• Streamlit Dashboard/Application (30 Oct 2025): View Live Application here
• Pitch deck / PPT (30 Oct 2025): View PPT
• Demo video submission record (5 Oct 2025): Watch here
• Idea Submission PPT (5 Oct 2025): View PPT

🧑‍💻 Author

Aravind Suresh
Data Engineer @ GE Aerospace | ML & Cloud Enthusiast

Abirami Sadasivam
SDE @ VISA | ML & Cloud Enthusiast

Sidhanth LS
Data Scientist @ Freshworks

🤝 Connect with the team on LinkedIn for collaborations!

---

🪶 License

This project is licensed under the MIT License.

---

🏁 Acknowledgments

Special thanks to:

• Snowflake for its developer ecosystem
• Alpha Vantage and News API for financial data sources
• Hackathon Mentors and collaborators for their support (Snowflake - The Dev Premier League)

---

⭐ If you like this project, give it a star on GitHub — your support keeps it growing!