ReelStream — HLS Video Streaming Service

Status: Early Stage / Experimental This is an exploration into video streaming at scale. It handles core HLS transcoding and streaming well, but it's not battle-tested for production edge cases (e.g., handling distributed locking on simultaneous identical uploads).

HLS streaming backend in Go. Built for high concurrency, can run locally or scale to cloud storage (S3) without code changes.

Stack

Layer	Technology
Language	Go 1.22+
HTTP	Fiber v2
Database	PostgreSQL 16 + pgx v5 + sqlc
Queue	Asynq (Redis-backed)
Cache	Redis 7
Storage	Local FS / MinIO / S3 (same interface)
Video	FFmpeg (transcode + HLS segment + thumbnail)
Auth	JWT RS256 + refresh tokens
Observability	zerolog + Prometheus + Jaeger (OpenTelemetry)

Quick Start

Prerequisites

• Go 1.22+
• Docker Desktop (for Postgres, Redis, MinIO)
• FFmpeg (winget install FFmpeg on Windows)

1. Clone & configure

git clone https://github.com/zyrridian/reelstream.git
cd reelstream
copy .env.example .env

2. Generate RSA keys (JWT)

go run cmd/genkeys/main.go

3. Start infrastructure

docker compose up -d --wait

4. Run database migrations

# Install migrate CLI (once):
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

migrate -path internal/db/migrations -database "postgres://reelstream:reelstream@localhost:5433/reelstream?sslmode=disable" up

5. Start the API server

go run ./cmd/api

6. Start the transcoding worker (separate terminal)

go run ./cmd/worker

API Endpoints

Method	Path	Auth	Description
POST	/api/v1/auth/register	—	Register
POST	/api/v1/auth/login	—	Login
POST	/api/v1/auth/refresh	—	Refresh token
POST	/api/v1/auth/logout	Bearer	Logout
POST	/api/v1/videos/upload	Bearer	Upload video
GET	/api/v1/videos/me	Bearer	My videos
GET	/api/v1/videos/:id	—	Get video
GET	/api/v1/videos/:id/status	Bearer	Processing status
DELETE	/api/v1/videos/:id	Bearer	Delete video
GET	/api/v1/feed	—	Global feed
GET	/api/v1/feed/me	Bearer	Personalised feed
GET	/api/v1/stream/:id/master.m3u8	—	HLS master playlist
GET	/api/v1/stream/:id/:quality/:file	—	HLS segment/playlist
GET	/api/v1/stream/:id/thumbnail.jpg	—	Thumbnail
GET	/health	—	Liveness probe
GET	/ready	—	Readiness probe
GET	/metrics	—	Prometheus metrics
GET	/docs/*	—	Swagger UI

HLS Pipeline

Upload → Validate → Store raw (MinIO) → Enqueue
    ↓
Worker: Download raw → Probe → Transcode
    ├── 1080p (5000kbps)  ─┐
    ├──  720p (2500kbps)   ├── HLS 2-second segments
    └──  360p  (800kbps)  ─┘
                            ↓
                    Upload segments → CDN bucket
                            ↓
                    Generate master.m3u8
                            ↓
                    Extract thumbnail
                            ↓
                    Mark PUBLISHED in PostgreSQL

Project Structure

├── cmd/
│   ├── api/             # API server entrypoint
│   └── worker/          # Transcoding worker entrypoint
├── internal/
│   ├── api/
│   │   ├── handlers/    # auth, upload, feed, stream, video
│   │   ├── middleware/  # JWT auth, rate limit, CORS, logger
│   │   └── routes/      # route registration
│   ├── cache/           # Redis wrapper + key constructors
│   ├── config/          # Viper config loader
│   ├── db/
│   │   ├── generated/   # sqlc models + queries (regenerate: make sqlc-generate)
│   │   ├── migrations/  # golang-migrate SQL files
│   │   └── queries/     # sqlc SQL source files
│   ├── ffmpeg/          # FFmpeg wrapper: probe, transcode, thumbnail
│   ├── observability/   # zerolog, Prometheus, health checks
│   ├── storage/         # StorageProvider interface + local/S3 implementations
│   └── worker/
│       ├── processor/   # Asynq server + handler registration
│       └── tasks/       # transcode, thumbnail, publish tasks
├── infra/               # Prometheus config
├── keys/                # RSA keypair (gitignored)
├── docker-compose.yml
├── Dockerfile.api
├── Dockerfile.worker
└── Makefile

Services (local)

Service	URL
API	http://localhost:8081
Swagger UI	http://localhost:8081/docs
Prometheus	http://localhost:9090
Jaeger UI	http://localhost:16686
MinIO Console	http://localhost:9001 (minioadmin/minioadmin)

Cloud Deployment

Ready for production? Set STORAGE_PROVIDER=s3, point to your AWS S3 bucket. Everything else stays the same—that's the whole point of the storage abstraction.

Tradeoffs & Limitations

• FFmpeg CPU Usage: Transcoding is extremely CPU-bound. If you upload a 4K 60fps video, the worker will aggressively consume available cores. In a real-world scenario, you'd scale the worker nodes completely independently from the API server.
• Single-Node Queue: Asynq works great for this scale, but if we needed to distribute transcoding across hundreds of global nodes, we might outgrow Redis and need a dedicated event streaming platform.
• No DRM: The HLS streams are currently unencrypted. Anyone with the .m3u8 link can download the raw video segments.