storage-service

Multi-provider S3-compatible storage service with presigned URL uploads, object metadata management, and service-to-service authentication.

Architecture

Client ──► storage-service ──► S3-compatible provider (MinIO, AWS S3, etc.)
                │
                └──► PostgreSQL (object metadata, providers, buckets, modules)

• Presigned URL upload — clients upload directly to the storage provider; the service never touches file bytes.
• Provider failover — multiple providers can be configured per module; failed uploads fall back to the next candidate.
• Soft delete — objects are marked deleted in the database; the actual file is removed from the provider.
• Integrity fields — ETag (server-side, from S3) and ChecksumSHA256 (client-provided) are set only on ConfirmUpload, never via Update.

Project Structure

├── api/                    OpenAPI 3.0 spec
├── cmd/
│   └── api/                Application entrypoint
├── config/                 YAML config (env-var overridable)
├── deploy/                 (optional — Kubernetes manifests)
├── docs/                   API docs (Redocly HTML)
├── internal/
│   ├── app/                App bootstrap (wire up deps)
│   ├── config/             Viper config loader
│   ├── health/             Liveness/readiness handlers
│   ├── platform/
│   │   ├── database/       PostgreSQL connection
│   │   ├── http/           Chi router, middleware (auth, metrics, rate-limit, CORS, logger)
│   │   └── logger/         Zap logger setup
│   └── storage/
│       ├── delivery/       HTTP handlers (providers, buckets, modules, objects)
│       ├── entity/         Domain models
│       ├── gateway/        S3 client abstraction
│       ├── readmodel/      Query projections
│       ├── repository/     PostgreSQL repositories (incl. integration tests)
│       ├── resolver/       Provider/bucket resolution
│       └── usecase/        Business logic + mocks
├── migrations/             golang-migrate SQL files
├── pkg/                    Shared utilities (encryption, errors, response, validation)
├── Dockerfile              Multi-stage build, non-root user
├── docker-compose.yml      App + PostgreSQL + migrate
└── sonar-project.properties

Endpoints

Method	Path	Auth	Description
GET	/health/live	—	Liveness probe
GET	/health/ready	—	Readiness probe (checks DB)
GET	/metrics	—	Prometheus metrics
GET	/api/docs	—	API documentation (HTML)
POST	/api/v1/storage/objects/upload	Token	Request presigned upload URL
POST	/api/v1/storage/objects/confirm	Token	Confirm upload & activate object
GET	/api/v1/storage/objects/{id}/download	Token	Get download URL
PATCH	/api/v1/storage/objects/{id}	Token	Update object metadata
DELETE	/api/v1/storage/objects/{id}	Token	Soft-delete object
GET	/api/v1/storage/buckets/{bucket_id}/objects	Token	List active objects by bucket
GET/POST/PUT/DELETE	/api/v1/storage/providers	Token	Provider CRUD
GET/POST/PUT/DELETE	/api/v1/storage/buckets	Token	Bucket CRUD
GET/POST/PUT/DELETE	/api/v1/storage/modules	Token	Module CRUD

Full spec: api/openapi.yaml

Prerequisites

• Go 1.26+
• PostgreSQL 17
• Docker (for integration tests)

Setup

# 1. Copy env
cp .env.example .env
# Edit .env with real values

# 2. Run database
docker compose up postgres -d

# 3. Run migrations
make migrate-up

# 4. Start service
make run

Configuration

Config is loaded from config/config.yaml with all secrets overridable via environment variables:

Env Var	Required	Default	Description
DATABASE_HOST	Yes	—	PostgreSQL host
DATABASE_PORT	No	5432	PostgreSQL port
DATABASE_USER	No	postgres	DB user
DATABASE_PASSWORD	Yes	—	DB password
DATABASE_NAME	No	storage_db	DB name
STORAGE_ENCRYPTION_KEY	Yes	—	32-byte hex for credential encryption
HRMS_SERVICE_TOKEN	Yes	—	Service auth token
COMMERCE_SERVICE_TOKEN	Yes	—	Service auth token
ADS_SERVICE_TOKEN	Yes	—	Service auth token
ADMIN_SERVICE_TOKEN	Yes	—	Service auth token
ADMIN_DASHBOARD_ORIGIN	No	—	CORS origin

Development

make run              # Start locally
make build            # Build binary
make test             # Unit tests + race detection
make test-integration # Integration tests (requires Docker)
make test-all         # Both
make lint             # golangci-lint
make mock             # Regenerate mocks (mockery)
make migrate-up       # Run pending migrations
make migrate-down     # Rollback 1 migration
make docker-build     # Build Docker image
make docker-run       # docker compose up --build
docker compose up     # Full stack (postgres + migrate + app)

Testing

• Unit tests: make test — pure Go tests with mocked repositories
• Integration tests: make test-integration — PostgreSQL via testcontainers-go (build tag: integration)
• Coverage focused on delivery + usecase packages (via sonar-project.properties exclusions)

CI/CD

GitHub Actions workflow (.github/workflows/ci.yml):

1. lint — go vet + golangci-lint
2. unit — go test -race -coverprofile=coverage.out ./...
3. integration — go test -tags=integration -race ./internal/storage/repository/postgres/...
4. build — go build + Docker image build (depends on all above)

Docker Compose for local deployment (PostgreSQL + migrations + app). See docker-compose.yml.

Security

• Non-root — Docker image runs as appuser (UID 1001)
• No hardcoded secrets — all credentials via environment variables (.env or K8s secrets)
• Fail-fast for misconfiguration — STORAGE_ENCRYPTION_KEY uses ${VAR:?error} syntax
• Service auth — per-service tokens validated on every API request
• Credential encryption — provider secrets encrypted at rest with AES (key from STORAGE_ENCRYPTION_KEY)

Object Lifecycle

  RequestUpload ──► pending ──► ConfirmUpload ──► active ──► Delete ──► deleted
                    (presigned   (set ETag,        (download,     (soft delete,
                     URL issued)  ChecksumSHA256)   update)        file removed)