Market Data Service

A real-time market data service that fetches, processes, and stores stock market data using FastAPI, PostgreSQL, and Kafka.

Features

• Real-time stock price data fetching
• Multiple data provider support (currently Alpha Vantage)
• Automated price polling at configurable intervals
• Data persistence in PostgreSQL
• Message queuing with Kafka
• RESTful API with Swagger documentation
• Database management with pgAdmin
• Docker containerization

Prerequisites

• Docker and Docker Compose
• Python 3.11+
• Alpha Vantage API key

Quick Start

1. Clone the repository:

git clone <repository-url>
cd market_data_service

2. Create a .env file in the project root:

DATABASE_URL=postgresql://postgres:postgres@db:5432/market_data
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
ALPHA_VANTAGE_API_KEY=your_api_key_here
APP_ENV=development
DEBUG=true
LOG_LEVEL=INFO

3. Start the services:

docker-compose up --build -d

4. Access the services:

• API Documentation: http://localhost:8000/docs
• pgAdmin: http://localhost:5050 (admin@admin.com/admin)

API Endpoints

Get Latest Price

GET /prices/latest?symbol=AAPL

Parameters:

• symbol: Stock symbol (e.g., AAPL, MSFT, GOOGL)
• provider: Data provider (default: alpha_vantage)

Start Price Polling

POST /prices/poll?symbols=["AAPL","MSFT"]&interval=60

Parameters:

• symbols: List of stock symbols
• interval: Polling interval in seconds

Database Management

1. Access pgAdmin at http://localhost:5050
2. Login with:
   • Email: admin@admin.com
   • Password: admin
3. Add a new server:
   • Name: Market Data
   • Host: db
   • Port: 5432
   • Database: market_data
   • Username: postgres
   • Password: postgres

Architecture

Components

• FastAPI: REST API framework
• PostgreSQL: Database for data persistence
• Kafka: Message queue for data processing
• pgAdmin: Database management interface
• Docker: Containerization

Data Flow

1. API receives request for stock data
2. Service fetches data from Alpha Vantage
3. Raw data is stored in PostgreSQL
4. Processed data is published to Kafka
5. Consumers process and store final data

Development

Project Structure

market_data_service/
├── alembic/              # Database migrations
│   └── migrations/      # Migration files
├── app/
│   ├── db/              # Database models and connection
│   ├── routers/         # API endpoints
│   ├── schemas/         # Pydantic models
│   ├── services/        # Business logic
│   └── main.py          # Application entry point
├── tests/               # Test suite
├── .env                 # Environment variables
├── docker-compose.yml   # Docker services
└── Dockerfile          # Application container

Running Tests

docker-compose run app pytest

Database Migrations

# Create a new migration
docker-compose run app alembic revision --autogenerate -m "description"

# Apply migrations
docker-compose run app alembic upgrade head

Troubleshooting

Common Issues

1. Database Connection Issues

   • Check if PostgreSQL container is running
   • Verify database credentials in .env
   • Check database logs: docker-compose logs db

2. API Not Responding

   • Check if FastAPI container is running
   • Verify API logs: docker-compose logs app
   • Check if Alpha Vantage API key is valid

3. Kafka Connection Issues

   • Check if Kafka and Zookeeper are running
   • Verify Kafka logs: docker-compose logs kafka

Logs

# View all logs
docker-compose logs -f

# View specific service logs
docker-compose logs -f app
docker-compose logs -f db
docker-compose logs -f kafka

Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.