Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

Real-Time ML Inference Service

A high-throughput event processing API that ingests streaming data, runs PyTorch model inference in real time, and computes per-user rolling-window statistics. Built with FastAPI and designed to handle thousands of requests per second.

What It Does

  1. Ingests event streams — accepts batches of user events, each containing a 3-feature vector and timestamp
  2. Runs ML inference — passes features through a PyTorch neural network to produce a score per event
  3. Tracks rolling statistics — maintains a 5-minute sliding window of scores per user and computes rolling medians on demand
  4. Reports latency metrics — tracks p50/p95/p99 latency for every endpoint

Architecture

                    ┌──────────────┐
  POST /ingest ───▶ │   FastAPI     │
                    │   Endpoints   │
  GET /users/…  ───▶│              │
  GET /stats    ───▶│              │
                    └──┬───┬───┬──┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │ Inference │ │  State   │ │ Metrics  │
        │ (PyTorch) │ │ Manager  │ │ Tracker  │
        └──────────┘ └──────────┘ └──────────┘

Key design decisions:

  • Protocol-based DI — core components (inference, state, metrics) are defined as Protocol interfaces, making them swappable without changing endpoint code
  • Sharded state management — user state is hash-partitioned across 16 shards with per-shard async locks, eliminating global lock contention
  • Lazy invalidation — medians are cached per user with a dirty flag; only recalculated on read after new writes
  • LRU eviction — bounded memory via OrderedDict-based LRU (configurable max users per shard)
  • Optional Ray Serve — inference can run locally or scale out via Ray Serve with automatic request batching (set USE_RAY=true)

API

Endpoint Method Description
/ingest POST Ingest a batch of events. Each event has user_id, timestamp, and features (3 floats). Runs inference and stores the score.
/users/{user_id}/median GET Returns the rolling 5-minute median inference score for a user.
/stats GET Returns total users, total events, median-of-medians across all users, and per-endpoint p50/p95/p99 latency.
/health GET Health check — confirms model is loaded.

Quick Start

# Install dependencies
make install

# Generate the PyTorch model
make model

# Start the service
make run

Or with Docker:

make docker-build
make docker-run

The service runs at http://localhost:8000.

Load Testing

The included event generator can simulate sustained high-throughput traffic:

python scripts/event_generator.py

Default: 5,000 events/sec across 10,000 users for 60 seconds using 50 concurrent threads. Use scripts/monitor_stats.sh to watch latency metrics during a load test.

Running Tests

# Start the service first
make run

# In another terminal
make test

Tests cover ingestion, per-user medians, rolling window expiration, concurrent users, and error handling.

Project Structure

app/
  main.py                  # FastAPI endpoints and request/response models
  model.py                 # PyTorch model loading and inference
  state.py                 # Rolling-window state with LRU eviction
  metrics.py               # Per-endpoint p50/p95/p99 latency tracking
  protocols.py             # Protocol interfaces for dependency injection
  dependencies.py          # Singleton wiring
  inference/
    local_inference.py     # Direct PyTorch inference (default)
    ray_deployment.py      # Ray Serve deployment with auto-batching
  state/
    sharded_state.py       # 16-shard async state manager
scripts/
  create_model.py          # Generate PyTorch model file
  event_generator.py       # Load testing — configurable RPS and duration
  monitor_stats.sh         # Live latency monitoring during load tests
tests/
  test_service.py          # Integration test suite

Tech Stack

  • FastAPI + Uvicorn — async HTTP with multi-worker support
  • PyTorch — neural network inference (3-feature input → score)
  • Pydantic — request/response validation
  • Ray Serve (optional) — distributed inference with automatic batching
  • Docker — containerized deployment

Configuration

Environment variables:

Variable Default Description
PORT 8000 Server port
MODEL_PATH inefficient_model.pt Path to PyTorch model file
MAX_USERS 10000 Max users in state (LRU eviction beyond this)
WINDOW_SECONDS 300 Rolling window size (seconds)
NUM_SHARDS 16 State manager shard count
USE_RAY false Enable Ray Serve for distributed inference

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages