Dagster Project

Dagster POC - Stock Market Data Pipeline

Architecture

dagster_project/
├── __init__.py           # Definitions entry point
├── assets/
│   └── stock_market.py   # Asset definitions
└── resources.py          # Resource configurations

Services

• Dagster Webserver: http://localhost:3001
• MinIO Console: http://localhost:9001
• Spark Master UI: http://localhost:8082
• Spark Worker UI: http://localhost:8081
• PostgreSQL: localhost:5433

Setup

# Build and start services
docker-compose up -d

# View logs
docker-compose logs -f dagster-webserver
docker-compose logs -f dagster-daemon

Stock Market Assets

• stock_prices: Fetch stock price data from API
• store_stock_prices: Store data to MinIO

Configuration

Resources are configured in resources.py:

• MinIO: 192.168.0.6:9000
• PostgreSQL: 192.168.0.5:5432