A high-throughput event processing API that ingests streaming data, runs PyTorch model inference in real time, and computes per-user rolling-window statistics. Built with FastAPI and designed to handle thousands of requests per second.
- Ingests event streams — accepts batches of user events, each containing a 3-feature vector and timestamp
- Runs ML inference — passes features through a PyTorch neural network to produce a score per event
- Tracks rolling statistics — maintains a 5-minute sliding window of scores per user and computes rolling medians on demand
- Reports latency metrics — tracks p50/p95/p99 latency for every endpoint
┌──────────────┐
POST /ingest ───▶ │ FastAPI │
│ Endpoints │
GET /users/… ───▶│ │
GET /stats ───▶│ │
└──┬───┬───┬──┘
│ │ │
┌────────┘ │ └────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Inference │ │ State │ │ Metrics │
│ (PyTorch) │ │ Manager │ │ Tracker │
└──────────┘ └──────────┘ └──────────┘
Key design decisions:
- Protocol-based DI — core components (inference, state, metrics) are defined as
Protocolinterfaces, making them swappable without changing endpoint code - Sharded state management — user state is hash-partitioned across 16 shards with per-shard async locks, eliminating global lock contention
- Lazy invalidation — medians are cached per user with a dirty flag; only recalculated on read after new writes
- LRU eviction — bounded memory via
OrderedDict-based LRU (configurable max users per shard) - Optional Ray Serve — inference can run locally or scale out via Ray Serve with automatic request batching (set
USE_RAY=true)
| Endpoint | Method | Description |
|---|---|---|
/ingest |
POST | Ingest a batch of events. Each event has user_id, timestamp, and features (3 floats). Runs inference and stores the score. |
/users/{user_id}/median |
GET | Returns the rolling 5-minute median inference score for a user. |
/stats |
GET | Returns total users, total events, median-of-medians across all users, and per-endpoint p50/p95/p99 latency. |
/health |
GET | Health check — confirms model is loaded. |
# Install dependencies
make install
# Generate the PyTorch model
make model
# Start the service
make runOr with Docker:
make docker-build
make docker-runThe service runs at http://localhost:8000.
The included event generator can simulate sustained high-throughput traffic:
python scripts/event_generator.pyDefault: 5,000 events/sec across 10,000 users for 60 seconds using 50 concurrent threads. Use scripts/monitor_stats.sh to watch latency metrics during a load test.
# Start the service first
make run
# In another terminal
make testTests cover ingestion, per-user medians, rolling window expiration, concurrent users, and error handling.
app/
main.py # FastAPI endpoints and request/response models
model.py # PyTorch model loading and inference
state.py # Rolling-window state with LRU eviction
metrics.py # Per-endpoint p50/p95/p99 latency tracking
protocols.py # Protocol interfaces for dependency injection
dependencies.py # Singleton wiring
inference/
local_inference.py # Direct PyTorch inference (default)
ray_deployment.py # Ray Serve deployment with auto-batching
state/
sharded_state.py # 16-shard async state manager
scripts/
create_model.py # Generate PyTorch model file
event_generator.py # Load testing — configurable RPS and duration
monitor_stats.sh # Live latency monitoring during load tests
tests/
test_service.py # Integration test suite
- FastAPI + Uvicorn — async HTTP with multi-worker support
- PyTorch — neural network inference (3-feature input → score)
- Pydantic — request/response validation
- Ray Serve (optional) — distributed inference with automatic batching
- Docker — containerized deployment
Environment variables:
| Variable | Default | Description |
|---|---|---|
PORT |
8000 |
Server port |
MODEL_PATH |
inefficient_model.pt |
Path to PyTorch model file |
MAX_USERS |
10000 |
Max users in state (LRU eviction beyond this) |
WINDOW_SECONDS |
300 |
Rolling window size (seconds) |
NUM_SHARDS |
16 |
State manager shard count |
USE_RAY |
false |
Enable Ray Serve for distributed inference |