dvc-datacatalog — DataCatalog

A self-hostable data catalog that integrates with DVC. It stores dataset metadata and .dvc pointer files (never the data itself), adds descriptions, tags, and access scopes, and lets teams discover datasets and pull them with plain dvc pull.

The catalog is the "index"; DVC + your existing remote (S3, Azure Blob, GCS, MinIO, SSH, …) remain the storage. A companion CLI, dvc-catalog, wraps dvc push/dvc pull so a single command publishes data to your remote and registers its metadata + .dvc snapshot in the catalog.

License: Apache-2.0 · Status: planning (see TODO.md) · Quality gate: local pre-commit; manual release automation only (see AGENTS.md).

What it does

Publish once, land twice. dvc-catalog push runs dvc push to your DVC remote, then uploads the resulting .dvc file(s), metadata, and the used project-level remote configuration to the catalog.
Describe & classify. Every dataset carries a description, tags, an owner, license, size/file-count, source Git repo + commit, and admin-definable custom key/value fields.
Govern access. Access is granted via groups and per-dataset scopes (read / write / admin). Groups can be synced from your IdP. Users can request access; owners/admins approve.
Discover. Combined full-text (name, description, README) and faceted search (tags, project, scope, custom fields), served by OpenSearch.
Consume. On a dataset page, copy the .dvc to clipboard, download it, or copy a dvc-catalog get <id> command — then dvc pull as usual.
Stay in sync with Git. Every published version records the Git commit SHA captured by the CLI, so a dataset version always points back at its source revision.
Deploy anywhere. First-class Docker images, a docker compose stack for local/dev, and a Helm chart (Gateway API, not Ingress) for Kubernetes.

Core concepts & domain model

Organization
  └── Project            (maps 1:1 to a Git repository)
        └── Dataset      (a tracked data artifact; has metadata + scopes)
              └── Version (immutable snapshot; content-hash + git commit SHA)
                    └── DvcArtifact (the .dvc file content + parsed outs/hashes)

Identity & access:
  User ── membership ──> Group ── grant ──> Scope(read|write|admin) on Dataset/Project
  AccessRequest (user -> dataset, pending|approved|denied)
  PersonalAccessToken (user -> API token for CLI/CI)

Cross-cutting:
  Tag, CustomFieldDefinition / CustomFieldValue
  AuditEvent (who did what, when, to which entity)
  Notification, NotificationChannel

Key entities

Entity	Purpose
`Organization`	Top-level tenant boundary.
`Project`	Bound 1:1 to a Git repo; owns datasets; sets default dataset visibility.
`Dataset`	Logical data artifact; holds metadata, tags, scopes, custom fields.
`Version`	Immutable snapshot created on each push; stores content/config hash + Git commit SHA + lineage + used DVC remote configuration.
`DvcArtifact`	The raw `.dvc` file text plus parsed `md5`/`outs`/size for that version.
`Group` / `Scope`	Authorization: groups hold members; scopes grant `read`/`write`/`admin`.
`AccessRequest`	Self-service access workflow with approval.
`AuditEvent`	Append-only record of every mutating action.

High-level architecture

        ┌───────────────────────┐                  ┌────────────────┐
        │      Web Browser      │                  │  dvc-catalog   │
        │ React + Vite + shadcn │                  │     CLI/CI     │
        └───────────┬───────────┘                  └───┬────────┬───┘
                    │ HTTPS (SPA + REST/OpenAPI)       │        │
                    ▼                                  │        │ dvc push/pull
     ┌────────────────────  Gateway API  ─────────────▼──┐     │  (never via
     │      (K8s Gateway / Traefik in compose, TLS)       │     │   the catalog)
     └───────────────────────┬────────────────────────────┘     │
                             │  REST + PAT                      │
                 ┌───────────▼────────────┐                     │
                 │      FastAPI API       │                     │
                 │   (uvicorn/gunicorn)   │                     │
                 │  REST + built SPA on   │                     │
                 │     a single port      │                     │
                 └──┬─────────┬────────┬──┘                     │
       auth         │         │ enqueue│ read/write             │
   ┌────────────────▼──┐  ┌───▼────┐  ┌▼─────────────┐  ┌───────▼────────┐
   │  Auth backends    │  │ Redis  │  │  PostgreSQL  │  │   DVC Remote   │
   │ OIDC/SAML/local   │  │ broker │  │ (source of   │  │ S3/Azure/GCS/  │
   └───────────────────┘  └───┬────┘  │   truth)     │  │  MinIO/SSH ... │
                              │       └──────┬───────┘  └────────────────┘
                       ┌──────▼───────┐      │ index from
                       │   Celery     │◀─────┘
                       │   workers    │
                       │ (indexing,   │
                       │  notify)     │
                       └──────┬───────┘
                              │ index / query
                       ┌──────▼───────┐
                       │  OpenSearch  │
                       │   (search)   │
                       └──────────────┘

Observability: OpenTelemetry traces + Prometheus metrics + structured JSON logs
               → Grafana / Tempo / Loki (dashboards shipped in the Helm chart).

Component responsibilities

FastAPI API — REST endpoints, OpenAPI schema, authn/authz, validation, writes to Postgres, enqueues Celery jobs, serves signed .dvc downloads, and serves the built frontend SPA on the same port (no separate web server).
Celery workers — (re)index datasets into OpenSearch, deliver notifications, run periodic maintenance jobs.
PostgreSQL — system of record for all metadata, .dvc text, versions, ACLs, audit log.
OpenSearch — denormalized search index (full-text + facets); rebuildable from Postgres at any time.
Redis — Celery broker/result backend and short-lived caches.
Frontend — SPA for browsing, searching, dataset detail, admin, access requests. Built to static assets and served by the FastAPI app itself, so the UI and the API share a single origin and port.
dvc-catalog CLI — client-side wrapper around DVC + catalog REST API.

Request/data flows

Publish (`dvc-catalog push`)

CLI validates auth (PAT or OIDC device flow) and resolves the project from .dvc/catalog (the committed catalog connection config).
CLI runs dvc push to the configured DVC remote (data leaves the client directly to the remote — never through the catalog).
CLI reads the .dvc file(s), their .dvc.meta sidecars (description, owner, tags, license, custom fields), and only the used remote sections from project-level .dvc/config. It never reads .dvc/config.local, global, or system DVC configuration.
CLI computes a hash over the pointer files and canonical remote snapshot, captures the current Git commit SHA, and POSTs the .dvc text, metadata, and storage configuration to /api/v1/datasets/{id}/versions. A push is refused if the catalog is not configured or a selected sidecar is incomplete; a remote available only outside project config is registered as unavailable rather than blocking publication.
API persists an immutable Version + DvcArtifact, writes an AuditEvent, enqueues an OpenSearch index job and notifications.

Discover & consume

User searches/browses in the SPA (queries hit OpenSearch via the API).
On the dataset page the user reviews the published DVC location/setup steps, copies the .dvc to clipboard, downloads it, or copies dvc-catalog get <dataset>@<version>.
User drops the .dvc into their repo (or runs dvc-catalog get) and runs dvc pull, which fetches data straight from the DVC remote.

Technology stack

Layer	Choice
Backend	Python 3.12, FastAPI, Pydantic v2, SQLAlchemy 2.0, Alembic
Async/jobs	Celery + Redis
Database	PostgreSQL (source of truth)
Search	OpenSearch (full-text + faceted)
Frontend	React + Vite + TypeScript, shadcn/ui + Tailwind, TanStack Query + Router
CLI	Python (Typer), wraps the `dvc` package
Auth	Pluggable: OIDC/Keycloak, SAML, local email+password; PATs for machines
Packaging	uv (backend/CLI) · pnpm (frontend)
Deploy	Docker, docker compose, Helm (Gateway API)
Observability	OpenTelemetry, Prometheus, structured logs, Grafana dashboards
Quality	pre-commit (ruff, mypy, bandit, detect-secrets, gitleaks, …), pytest, Vitest, Playwright

Repository structure

dvc-datacatalog/
├── AGENTS.md                     # Mandatory working rules (pre-commit, no ignores)
├── README.md                     # This file
├── TODO.md                       # Detailed 10-task delivery plan
├── LICENSE                       # Apache License 2.0
├── NOTICE                        # Project and DVC attribution notices
├── .pre-commit-config.yaml       # The local development quality gate
├── .github/workflows/release.yaml # Manual, independently published releases
├── .secrets.baseline             # detect-secrets baseline (reviewed)
├── pyproject.toml                # Shared Python tool config (ruff/mypy/bandit/codespell)
│
├── backend/                      # FastAPI service
│   ├── app/
│   │   ├── main.py               # App factory, router mounting, OTel setup
│   │   ├── api/v1/               # Routers: datasets, versions, search, auth, admin
│   │   ├── core/                 # Settings, security, logging, dependencies
│   │   ├── auth/                 # Pluggable providers: oidc/, saml/, local/, pat/
│   │   ├── models/               # SQLAlchemy models
│   │   ├── schemas/              # Pydantic request/response models
│   │   ├── services/             # Domain logic (catalog, acl, search, notify)
│   │   ├── search/               # OpenSearch client, mappings, indexers
│   │   ├── tasks/                # Celery app + task modules
│   │   └── db/                   # Session, base, seed
│   ├── alembic/                  # Migrations
│   ├── tests/                    # pytest (unit + integration)
│   ├── Dockerfile
│   └── pyproject.toml            # uv-managed deps
│
├── frontend/                     # React + Vite SPA (built into the API image)
│   ├── src/
│   │   ├── app/                  # Router, providers, layout
│   │   ├── components/ui/        # shadcn/ui primitives (generated)
│   │   ├── features/             # search, datasets, projects, admin, auth, access-requests
│   │   ├── lib/                  # API client (generated from OpenAPI), hooks, utils
│   │   └── styles/               # Tailwind
│   ├── tests/                    # Vitest
│   ├── e2e/                      # Playwright
│   └── package.json              # pnpm
│
├── cli/                          # `dvc-catalog`
│   ├── dvc_catalog/
│   │   ├── __main__.py           # Typer entrypoint
│   │   ├── commands/             # init, login, whoami, status, push, get, pull
│   │   ├── repo.py               # Locate .dvc repo; discover .dvc files
│   │   ├── config.py             # Read/write .dvc/catalog + .dvc/catalog.local
│   │   ├── metadata.py           # Load/validate .dvc.meta sidecars
│   │   ├── api_client.py         # REST client + PAT/OIDC auth
│   │   └── dvc_bridge.py         # Thin wrapper over the dvc CLI
│   ├── tests/
│   └── pyproject.toml            # uv; exposes `dvc-catalog` console script
│
├── deploy/
│   ├── compose/                  # docker-compose.yml + overrides + Traefik(Gateway)
│   ├── helm/dvc-datacatalog/     # Chart: api (serves SPA), worker, gateway, HPA, dashboards
│   └── images/                   # Shared base images / entrypoints
│
└── docs/                         # Architecture, ADRs, runbooks, API guides

Authentication & authorization

Authentication is pluggable and chosen at deploy time via configuration. Supported providers (one or more can be enabled simultaneously):

OIDC / Keycloak — Authorization Code + PKCE for the SPA; device flow for the CLI; IdP groups mapped to catalog groups.
SAML 2.0 — for enterprises standardized on SAML SSO.
Local email + password — self-contained, with Argon2 hashing and TOTP MFA.
Personal Access Tokens (PATs) — for the CLI and CI; scoped, expiring, revocable.

Authorization model: groups + per-dataset scopes (read / write / admin), grantable to users or groups. When the IdP supplies groups, scopes can be synced automatically. Default dataset visibility is configurable per project. A self-service access-request workflow lets a user request read/write; owners/admins approve, which auto-grants the scope.

Search

OpenSearch powers a combined experience:

Full-text over name, description, and README with ranking and highlighting.
Faceted filters over tags, project, organization, scope/visibility, license, and custom fields.
Autocomplete/typeahead for tags and dataset names.

Postgres remains the source of truth; the index is fully rebuildable via a Celery reindex task. Access filtering is applied so results respect the caller's scopes.

Versioning model

Each dvc-catalog push creates an immutable Version identified by the content hash of the .dvc outputs and the location-bearing subset of its canonical project-level remote snapshot (remote names and URLs). Moving unchanged objects to another published remote URL therefore records a new version; changing a tuning knob such as jobs does not.
Every version also stores the Git commit SHA (as metadata) plus basic lineage (source repo/commit, produced-by pipeline stage when available).
Full history is browsable; you can fetch any historical .dvc and dvc pull that exact snapshot.

The `dvc-catalog` CLI

A standalone wrapper (no fork of DVC required) that is configured the DVC way: all catalog settings live under the repo's .dvc/ directory, and dataset metadata lives in a YAML sidecar next to each .dvc file. There are deliberately no metadata flags — publishing is driven entirely by committed config, so a push is reproducible and reviewable.

dvc-catalog init                      # scan repo for .dvc files, scaffold config + sidecars
dvc-catalog login                     # store a PAT in .dvc/catalog.local (gitignored)
dvc-catalog whoami                    # show configured url / project / auth state
dvc-catalog status                    # list every .dvc and whether its sidecar is complete
dvc-catalog push [targets]            # validate, run `dvc push`, then register metadata
dvc-catalog get <dataset>[@<version>] # writes the .dvc into your project
dvc-catalog pull <dataset>            # get + dvc pull in one step

Configuration layout

File	Committed?	Purpose
`.dvc/catalog`	yes (INI)	Catalog connection: `[catalog]` `url` + `project`.
`.dvc/catalog.local`	no (gitignored)	Secrets: `[auth]` `token`. Written by `login`.
`<path>.dvc.meta`	yes (YAML)	Per-dataset metadata next to each `.dvc` file.

The token is kept in a dedicated .dvc/catalog.local (added to .dvc/.gitignore) rather than DVC's own config.local, so it never trips DVC's config validation.

A metadata sidecar looks like this (data/images.dvc → data/images.dvc.meta):

description: "Curated training images"   # required
owner: "team-vision"                     # required
tags: [images, cv]
license: MIT
custom_fields:
  sensitivity: low

How `push` behaves

Init first. push fails if .dvc/catalog has no url/project or no token is stored — run dvc-catalog init and dvc-catalog login first.
Multiple .dvc files are first-class. With no targets, every .dvc in the repo is selected (mirroring dvc push); pass targets to restrict the set.
Metadata is required. Every selected .dvc must have a complete sidecar (description + owner); otherwise the push aborts and lists the offenders. Use dvc-catalog status to check readiness and --dry-run to validate without pushing.

The CLI never uploads data to the catalog — only .dvc text + metadata.

Deployment

All three paths are first-class:

Docker — multi-stage images for api (which bundles the built SPA), worker, and cli.
docker compose — one-command local stack: api, worker, Postgres, Redis, OpenSearch, and a Traefik Gateway with TLS.
Helm (deploy/helm/dvc-datacatalog) — Deployments for api/worker, Kubernetes Gateway API resources (Gateway + HTTPRoute, not Ingress), HPAs, PodDisruptionBudgets, values.yaml toggles for each auth backend, external vs bundled Postgres/Redis/OpenSearch, and Grafana dashboards.

TLS is terminated at the Gateway in every environment. The SPA and the API are served by the same FastAPI process on a single port, so no separate web server (nginx or otherwise) is deployed.

Observability

Traces: OpenTelemetry across API and Celery, exported via OTLP (Tempo/ Jaeger compatible).
Metrics: Prometheus (/metrics) for API traffic and process health.
Logs: structured JSON with correlation/trace IDs (Loki-friendly).
Dashboards: Grafana dashboards shipped in the Helm chart.

Local development

# 1. Clone and install the quality gate (mandatory — see AGENTS.md)
pip install pre-commit && pre-commit install --install-hooks

# 2. Backend
cd backend && uv sync && uv run alembic upgrade head
uv run uvicorn app.main:app --reload

# 3. Frontend (dev server with HMR, proxying API paths to the backend)
cd frontend && pnpm install && pnpm dev
# For a production-like run, `pnpm build`, then start the backend with
# APP_FRONTEND_DIST=../frontend/dist so it serves the SPA and the API
# from the same port.

# 4. Full stack
docker compose -f deploy/compose/compose.yaml up --build

# 5. Before every push
pre-commit run --all-files

There is no automatic push or pull-request CI. pre-commit remains the local quality gate. The manually dispatched release workflow repeats it before publishing artifacts.

Configuration reference

Configuration is environment-variable driven (12-factor). Every setting is read from APP_<FIELD> — APP_DB_DSN, APP_AUTH_ENABLED_PROVIDERS, and so on. For the deployment groups below the unprefixed spelling (DB_DSN, AUTH_ENABLED_PROVIDERS, …) is accepted as a fallback; when both are set the APP_-prefixed one wins.

Prefix	Purpose
`APP_`	App name, base URL, environment, log level
`APP_FRONTEND_DIST`	Directory of the built SPA to serve on the API port; unset = API-only
`APP_DB_` / `DB_`	PostgreSQL DSN, pool sizing
`APP_REDIS_` / `REDIS_`	Broker/result URL
`APP_SEARCH_` / `SEARCH_`	OpenSearch host, index prefix
`APP_AUTH_` / `AUTH_`	Enabled providers + per-provider settings (`AUTH_OIDC_`, `AUTH_SAML_`, `AUTH_LOCAL_*`)
`APP_MAIL_` / `NOTIFY_`	SMTP host, port, credentials, envelope sender
`APP_OTEL_` / `OTEL_`	OTLP endpoint, sampling

APP_AUTH_ENABLED_PROVIDERS accepts either spelling: APP_AUTH_ENABLED_PROVIDERS=local,oidc or '["local","oidc"]'.

Redis is required in production

Redis-backed credential throttling and Celery indexing are opt-in, so upgrading an existing deployment does not silently acquire a new hard dependency: with APP_REDIS_RATE_LIMIT_ENABLED on and Redis unreachable, every credential endpoint answers 503 and authentication stops working entirely.

Production refuses to start without both. APP_ENVIRONMENT=production requires APP_REDIS_RATE_LIMIT_ENABLED=true and APP_CELERY_ENABLED=true, and names the missing variable in the startup error rather than leaving an operator to discover a per-worker rate limit or a permanently stale search index in production. Provision Redis and OpenSearch, and run a worker and beat, before enabling them.

Settings that must be chosen deliberately

Variable	Default	Why it matters
`APP_AUTH_LOCAL_REGISTRATION_ENABLED`	`false`	Self-registration is opt-in. Enabling it requires `APP_AUTH_DEFAULT_ORGANIZATION_SLUG`.
`APP_AUTH_DEFAULT_ORGANIZATION_SLUG`	(unset)	The tenant new accounts join. Membership is never inferred from row order.
`APP_AUTH_RESET_DELIVERY`	`log`	`log` prints the reset token (development only); production requires `smtp`.
`APP_AUTH_COOKIE_SECURE`	`true`	Production refuses to send session cookies over plain HTTP.
`APP_AUTH_MAX_FAILED_LOGINS`	`5`	Failed attempts before the account locks.
`APP_AUTH_LOCKOUT_SECONDS`	`900`	Base lockout interval; doubles for each further failure.
`APP_AUTH_RATE_LIMIT_MAX_ATTEMPTS`	`30`	Attempts permitted per sliding window. Shared across API workers only when `APP_REDIS_RATE_LIMIT_ENABLED` is on.
`APP_REDIS_BROKER_URL`	`redis://localhost:6379/0`	Celery broker used by API dispatchers, workers, and beat.
`APP_REDIS_RESULT_BACKEND`	`redis://localhost:6379/1`	Celery task result backend.
`APP_REDIS_RATE_LIMIT_URL`	`redis://localhost:6379/2`	Shared credential-throttling state.
`APP_REDIS_RATE_LIMIT_ENABLED`	`false`	Requires Redis. Shares the credential throttle across API workers; without it each worker counts separately. Required in production.
`APP_CELERY_ENABLED`	`false`	Requires Redis. Publishes post-commit catalog index events to Celery; without it the search index is never updated. Required in production.
`APP_CELERY_CONSISTENCY_INTERVAL_SECONDS`	`900`	Beat interval for fingerprint-based PostgreSQL/OpenSearch drift repair.
`APP_SEARCH_URL`	`http://localhost:9200`	OpenSearch endpoint used by API and worker processes.
`APP_SEARCH_INDEX_PREFIX`	`dvc-catalog`	Prefix for concrete dataset indexes and the live alias.
`APP_AUTH_SESSION_RETENTION_DAYS`	`90`	Days to retain revoked or long-expired session metadata before pruning.
`APP_AUTH_PASSWORD_RESET_RETENTION_DAYS`	`30`	Days to retain consumed or long-expired reset-token hashes before pruning.
`APP_TRUST_PROXY_HEADERS`	`false`	Enable only behind a proxy that always sets `X-Forwarded-For`, so the audit trail records the client and not the gateway.
`APP_SEED_ADMIN_PASSWORD`	(unset)	Password for a newly seeded admin. Required when seeding a production database.

The dvc-catalog CLI is configured per-repository instead of via env vars: .dvc/catalog (committed: url, project), .dvc/catalog.local (gitignored: token), and a <path>.dvc.meta YAML sidecar per dataset. See The dvc-catalog CLI.

Security & engineering practices

Local quality gate, no automatic CI. Every commit must pass pre-commit run --all-files; the manual release workflow repeats the gate before publication (see AGENTS.md).
No inline suppressions of any kind (# type: ignore, # noqa, # nosec, eslint-disable, @ts-ignore, secret allow-lists, …). Fix the root cause.
Secrets scanned by detect-secrets + gitleaks; never commit real secrets.
Typed & tested: mypy --strict, pytest (backend), Vitest + Playwright (frontend).
Data never in Git: large files blocked by pre-commit; only .dvc + metadata are tracked.

License and DVC attribution

DVC Data Catalog is licensed under the Apache License, Version 2.0. It integrates with DVC (Data Version Control), an independent project maintained by Treeverse and also distributed under Apache-2.0. See NOTICE for the DVC attribution and upstream license reference.

See TODO.md for the detailed, step-by-step delivery plan.

Navigation Menu

Repository files navigation

dvc-datacatalog — DataCatalog

Table of contents

What it does

Core concepts & domain model

Key entities

High-level architecture

Component responsibilities

Request/data flows

Publish (`dvc-catalog push`)

Discover & consume

Technology stack

Repository structure

Authentication & authorization

Search

Versioning model

The `dvc-catalog` CLI

Configuration layout

How `push` behaves

Deployment

Observability

Local development

Configuration reference

Redis is required in production

Settings that must be chosen deliberately

Security & engineering practices

License and DVC attribution

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.agents/skills		.agents/skills
.github/workflows		.github/workflows
backend		backend
cli		cli
deploy		deploy
docs		docs
frontend		frontend
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.markdownlint.yaml		.markdownlint.yaml
.pre-commit-config.yaml		.pre-commit-config.yaml
.secrets.baseline		.secrets.baseline
.yamllint.yaml		.yamllint.yaml
AGENTS.md		AGENTS.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
TODO.md		TODO.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation

dvc-datacatalog — DataCatalog

Table of contents

What it does

Core concepts & domain model

Key entities

High-level architecture

Component responsibilities

Request/data flows

Publish (dvc-catalog push)

Discover & consume

Technology stack

Repository structure

Authentication & authorization

Search

Versioning model

The dvc-catalog CLI

Configuration layout

How push behaves

Deployment

Observability

Local development

Configuration reference

Redis is required in production

Settings that must be chosen deliberately

Security & engineering practices

License and DVC attribution

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Publish (`dvc-catalog push`)

The `dvc-catalog` CLI

How `push` behaves