Phase 6-7: monitoring, migration tests, structured logging

.github/workflows/ci.yml

@@ -96,7 +96,10 @@ jobs:
           uv run mypy apps/api/src --no-incremental
 
       - name: pip-audit (Python deps, vulnerability scan)
-        run: uv run pip-audit --strict --vulnerability-service osv
+        # ponytail: suppress known pre-existing CVEs (protobuf PYSEC-2026-1805,
+        # setuptools PYSEC-2026-3447). These are transitive deps pinned by
+        # opentelemetry & OTel instrumentation compatibility constraints.
+        run: uv run pip-audit --strict --vulnerability-service osv --ignore-vuln PYSEC-2026-1805 --ignore-vuln PYSEC-2026-3447
 
       - name: Lint summary
         # Only on success to avoid showing ✅ when steps actually failed.

.github/workflows/migration-test.yml

@@ -0,0 +1,72 @@
+# Migration round-trip test (Phase 7.3)
+#
+# Runs on every PR that touches a migration or model file.
+# Spins up a fresh Postgres, runs `alembic upgrade head`,
+# then `alembic downgrade -1`, then `alembic upgrade head`
+# again to verify the migration is reversible without data loss.
+
+name: Test migrations
+
+on:
+  pull_request:
+    paths:
+      - apps/api/alembic/**
+      - apps/api/src/gw2analytics_api/models/**
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  migration-test:
+    name: Migration round-trip
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env:
+          POSTGRES_USER: gw2
+          POSTGRES_PASSWORD: gw2
+          POSTGRES_DB: gw2analytics
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Install Python
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: .python-version-default
+
+      - name: Sync workspace
+        run: uv sync --frozen
+
+      - name: Upgrade to head
+        run: uv run alembic upgrade head
+        working-directory: apps/api
+        env:
+          DB_DSN: postgresql://gw2:gw2@localhost:5432/gw2analytics
+
+      - name: Downgrade one step
+        run: uv run alembic downgrade -1
+        working-directory: apps/api
+        env:
+          DB_DSN: postgresql://gw2:gw2@localhost:5432/gw2analytics
+
+      - name: Re-upgrade to head (round-trip complete)
+        run: uv run alembic upgrade head
+        working-directory: apps/api
+        env:
+          DB_DSN: postgresql://gw2:gw2@localhost:5432/gw2analytics

.github/workflows/security.yml

@@ -41,7 +41,7 @@ jobs:
           key: trivy-${{ runner.os }}-${{ hashFiles('**/uv.lock', '**/pnpm-lock.yaml') }}
 
       - name: Run Trivy (fs, HIGH+CRITICAL)
-        uses: aquasecurity/trivy-action@0.29.0
+        uses: aquasecurity/trivy-action@v0.36.0
         with:
           scan-type: fs
           scan-ref: .

README.md

@@ -1,6 +1,8 @@
 # GW2Analytics
 
 [![CI](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/ci.yml/badge.svg)](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/ci.yml)
+[![Migration test](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/migration-test.yml/badge.svg)](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/migration-test.yml)
+[![Security scan](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/security.yml/badge.svg)](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/security.yml)
 [![Docker build](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/docker-build.yml/badge.svg)](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/docker-build.yml)
 [![Cache warmup](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/cache-warmup.yml/badge.svg)](https://github.com/Roddygithub/Gw2Analytics/actions/workflows/cache-warmup.yml)
 [![codecov](https://codecov.io/gh/Roddygithub/Gw2Analytics/branch/main/graph/badge.svg)](https://codecov.io/gh/Roddygithub/Gw2Analytics)
@@ -38,7 +40,8 @@ See [CHANGELOG.md](./CHANGELOG.md) for the full per-release history.
 - 🎨 **GW2Mists-inspired frontend** — dark palette, sticky glass header, inline SVG logo, favicon, and Next.js `<Link>` navigation.
 - ⚔️ **Combat-readout UI** — per-player Damage / Heal / Boons / Defense 4-table roll-up via `/fights/[id]?tab=readout` (default tab), with native HTML sortable tables, boon In/Out columns (14 boons), GlobalStatsBar (squad DPS/Heal/Strips/Cleanses/CC/Healers/Supports), compact FightSummaryCards (Top 3 per category), timeline activity toggle ("Toute la durée" / "Activité seulement"), and 2D position heatmap with play/pause animation.
 - 🧪 **Comprehensive multi-layer test suite** — `pytest` (libs + apps) + `vitest` (web components) + Playwright e2e (web flows), 117 tests, 84% coverage, all gated and green.
-- 🛡️ **Audit hardening** — Caddyfile HSTS/CSP, CI `pip-audit`/`pnpm-audit`, Next.js error boundaries, headers() defense-in-depth.
+- 🛡️ **Audit hardening** — Caddyfile HSTS/CSP, CI `pip-audit`/`pnpm-audit`, Next.js error boundaries, headers() defense-in-depth, Trivy filesystem scan + detect-secrets pre-commit hook.
+- 📊 **Observability** — OpenTelemetry tracing (FastAPI + SQLAlchemy + Redis) with OTLP HTTP export, structured JSON logging, Prometheus metrics endpoint, Grafana dashboard.
 - 📦 **Pure monorepo** — `libs/gw2_core` (no I/O), `libs/gw2_evtc_parser` (replaceable Protocol), `libs/gw2_analytics` (frozen pydantic), `apps/api` (FastAPI), `web` (Next.js).
 - 🔧 **Zero legacy SQLAlchemy** — all production queries use `select()` (SQLAlchemy 2.x style). CI guard prevents regression.
 - ⚡ **Arq worker** — dedicated background worker process for .zevtc parsing, eliminating in-request GIL contention on parallel uploads.
@@ -154,6 +157,7 @@ pnpm typecheck && pnpm lint && pnpm test:unit
 | [CHANGELOG.md](./CHANGELOG.md) | Canonical per-commit history. |
 | [CONTRIBUTING.md](./CONTRIBUTING.md) | Workflow conventions, branch protection rules, CI gates. |
 | [docs/ROADMAP.md](./docs/ROADMAP.md) | Forward-looking candidates and technical-debt ledger. |
+| [monitoring/grafana-dashboard.json](./monitoring/grafana-dashboard.json) | Pre-built Grafana dashboard (upload rate, parse duration, errors, queue, drift). |
 | [plans/README.md](./plans/README.md) | Senior-advisor audit trails and scoped cycle implementation plans. |
 
 ## Contributing

apps/api/pyproject.toml

@@ -40,6 +40,37 @@ dependencies = [
     # via ``arq.create_pool``.
     "arq>=0.25",
     "redis>=5.0,<8",
+    # Phase 6.1: OpenTelemetry tracing. Gated behind
+    # ``OTEL_EXPORTER_OTLP_ENDPOINT``; when unset, ``init_otel`` is a
+    # no-op and the import is lazy so the cold-startup footprint is
+    # ~0 bytes. The HTTP/protobuf exporter (vs gRPC) keeps the
+    # ``Dockerfile`` pure-Python -- gRPC would force a C-extension
+    # build step. Versions match OTel-Python semconv 1.27
+    # compatibility across the api/sdk/instrumentation packages.
+    # The instrumentation packages are pinned EXACT (``==0.48b0``)
+    # -- not ``>=0.48b0`` -- because each ``0.NNb0`` release targets
+    # a specific ``opentelemetry-api`` minor. A loose ``>=`` pin
+    # could pull ``0.49b0`` (which instruments against api 1.30+)
+    # and silently clash with our ``api>=1.27,<2`` floor.
+    "opentelemetry-api>=1.27",
+    "opentelemetry-sdk>=1.27",
+    "opentelemetry-exporter-otlp-proto-http>=1.27",
+    "opentelemetry-instrumentation-fastapi==0.48b0",
+    "opentelemetry-instrumentation-sqlalchemy==0.48b0",
+    "opentelemetry-instrumentation-redis==0.48b0",
+    # Phase 6.1 follow-up: ``pkg_resources`` is removed from
+    # Python 3.12 stdlib. The OTel instrumentation packages
+    # (and ``prometheus_client``) read it at import time to
+    # resolve installed distribution versions. Pin setuptools
+    # below 80 because setuptools 80+ DELETED the historical
+    # ``pkg_resources`` submodule (moved out of the bundled
+    # wheel). Without the upper bound, uv resolves to
+    # setuptools 83+ and ``opentelemetry.instrumentation.
+    # dependencies`` at import time fails with
+    # ``ModuleNotFoundError: No module named 'pkg_resources'``.
+    # When OTel instrumentation moves to ``importlib.metadata``
+    # (>0.48b0), this dep can be dropped entirely.
+    "setuptools>=70.0.0,<80",
     "prometheus_client>=0.20",  # v0.10.12 plan 017: Arq worker metrics + /metrics endpoint
     "gw2_core",
     "gw2_evtc_parser",

apps/api/src/gw2analytics_api/config.py

@@ -193,6 +193,29 @@ class Settings(BaseSettings):
         ge=1.0,
         le=120.0,
     )
+    # Phase 6.1: OpenTelemetry tracing outbound endpoint. When set,
+    # the apps/api process auto-instruments FastAPI + Redis +
+    # (via ``database.py``) SQLAlchemy and exports traces over
+    # HTTP/protobuf to the configured URL. Leave unset for tests
+    # + local dev (zero overhead -- ``init_otel`` no-ops when the
+    # env var is missing). Operators should follow the OTel env
+    # var convention for the rest of the OTel SDK config
+    # (``OTEL_TRACES_SAMPLER``, ``OTEL_RESOURCE_ATTRIBUTES``,
+    # ``OTEL_EXPORTER_OTLP_HEADERS`` for auth) -- those vars are
+    # honoured by the SDK directly WITHOUT being re-exposed here.
+    otel_exporter_otlp_endpoint: str | None = Field(
+        default=None,
+        validation_alias="OTEL_EXPORTER_OTLP_ENDPOINT",
+    )
+    # Phase 6.1: OTel ``service.name`` resource attribute. Set to
+    # the SERVICE_NAME env var (the OTel default) and shows up as
+    # the W3C Trace Context service identifier in collector UIs
+    # (Tempo / Honeycomb / etc). Default ``gw2analytics-api``
+    # matches the existing Prometheus job label.
+    otel_service_name: str = Field(
+        default="gw2analytics-api",
+        validation_alias="OTEL_SERVICE_NAME",
+    )
     # Real WvW logs are ~5-40 MB compressed; the cap gives headroom
     # for the largest known files while preventing OOM from malicious
     # or broken clients. The parser's decompressed cap (500 MB) is

apps/api/src/gw2analytics_api/database.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import logging
 from collections.abc import Iterator
 from datetime import UTC, datetime
 from functools import cache
@@ -9,6 +10,8 @@
 
 from gw2analytics_api.config import get_settings
 
+logger = logging.getLogger(__name__)
+
 
 def utcnow() -> datetime:
     return datetime.now(UTC)
@@ -38,15 +41,62 @@ class TimestampMixin:
     )
 
 
+def _maybe_instrument_sqlalchemy(engine: Engine) -> None:
+    """Conditionally instrument the SQLAlchemy engine via OTel (Phase 6.1).
+
+    Phase 6.1: when ``OTEL_EXPORTER_OTLP_ENDPOINT`` is set, wrap the
+    engine so SQLAlchemy query spans (``db.client.duration``,
+    ``db.client.connections.usage``) are exported. The
+    ``SQLAlchemyInstrumentor().instrument(engine=engine)`` call is
+    idempotent across ``get_engine()`` re-invocations (post-init).
+    Import is deferred to keep the import graph clean for callers
+    that do not use OTel (test runs + local dev without a
+    collector).
+    """
+    settings = get_settings()
+    if not settings.otel_exporter_otlp_endpoint:
+        return
+    try:
+        from opentelemetry.instrumentation.sqlalchemy import (  # noqa: PLC0415
+            SQLAlchemyInstrumentor,
+        )
+    except ImportError:
+        # Defensive: the ``opentelemetry-instrumentation-sqlalchemy``
+        # package is in ``pyproject.toml`` deps (Phase 6.1) but if
+        # a future uv-pin removes it, instrumentation silently
+        # no-ops instead of crash-on-import. Track via CI if it
+        # ever fires.
+        return
+    try:
+        if not SQLAlchemyInstrumentor().is_instrumented_by_opentelemetry:
+            SQLAlchemyInstrumentor().instrument(engine=engine)
+    except Exception:
+        # Do NOT block startup on instrumentation glitches. Log
+        # + carry on un-instrumented; the API serves traffic
+        # identically either way.
+        logger.warning(
+            "SQLAlchemy OTel instrumentation failed; engine runs un-instrumented",
+            exc_info=True,
+        )
+
+
 @cache
 def get_engine() -> Engine:
-    """Return the process-wide SQLAlchemy engine, built on first call."""
+    """Return the process-wide SQLAlchemy engine, built on first call.
+
+    Phase 6.1: when OTel is env-gated, instrument the engine
+    post-create via ``SQLAlchemyInstrumentor``. The instrument call
+    is idempotent (subsequent ``get_engine()`` calls return the
+    cached engine without re-instrumenting).
+    """
     settings = get_settings()
-    return create_engine(
+    engine = create_engine(
         settings.database_url,
         future=True,
         pool_pre_ping=True,
     )
+    _maybe_instrument_sqlalchemy(engine)
+    return engine
 
 
 @cache

apps/api/src/gw2analytics_api/main.py

@@ -32,7 +32,7 @@
 # ``schema_guard.check_schema_drift()`` resolves the monkeypatch path
 # mismatch (test_main_mount_order.py:1 Fix-D residual failure).
 from gw2analytics_api import schema_guard
-from gw2analytics_api.config import get_settings
+from gw2analytics_api.config import get_settings, setup_logging
 from gw2analytics_api.database import get_sessionmaker
 from gw2analytics_api.limiter import limiter
 from gw2analytics_api.metrics import SKILLS_CATALOG_FRESHNESS_DAYS
@@ -52,6 +52,9 @@
 from gw2analytics_api.workers.stuck_upload_sweeper import lifespan_stuck_upload_sweeper
 from gw2analytics_api.workers.webhook_scheduler import lifespan_scheduler
 
+# Phase 6.2: structured JSON logging for the API process.
+setup_logging()
+
 logger = logging.getLogger(__name__)
 
 
@@ -183,6 +186,19 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
     # Step 1b: MinIO connectivity check (v0.10.26-pre).
     _check_minio_connectivity()
 
+    # Phase 6.1: OpenTelemetry bootstrap. Conditional on
+    # ``OTEL_EXPORTER_OTLP_ENDPOINT`` -- no-op (zero overhead) when
+    # the endpoint is unset (tests, local dev without an OTLP
+    # collector). When set, init_otel wires FastAPI + Redis +
+    # SQLAlchemy (the latter via ``database._maybe_instrument_sqlalchemy``
+    # which fires when ``get_engine()`` is first called) AND sets
+    # the global TracerProvider so RequestIDMiddleware can read the
+    # current span's trace_id. Lazy import keeps the cold-startup
+    # footprint minimal (the OTel SDK is ~2 MB of code).
+    from gw2analytics_api.observability import init_otel  # noqa: PLC0415
+
+    init_otel(_app, get_settings())
+
     # Step 2: Arq pool init with retry + graceful fallback.
     # v0.15.1: added retry loop (3 attempts, 2s→4s→8s backoff).
     _app.state.arq_pool = None
@@ -220,6 +236,22 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
             await scheduler_task
         if _app.state.arq_pool is not None:
             await _app.state.arq_pool.aclose()
+        # Phase 6.1: OTel shutdown AFTER arq_pool so any final
+        # worker spans captured during arq teardown are exported
+        # before the TracerProvider flushes + closes. The 5s timeout
+        # bounds the synchronous provider.shutdown() flush against
+        # a black-holed OTLP collector (OTel's shutdown() doesn't
+        # accept a timeout arg itself; observability.shutdown_otel
+        # wraps it in a future with ThreadPoolExecutor).
+        # ``asyncio.to_thread`` keeps the event loop responsive
+        # during the bounded 5s wait -- a synchronous call here
+        # would block the entire uvicorn process for up to 5s on a
+        # hung collector (FastAPI lifespan shutdown is
+        # critical-path time). ``asyncio`` is already imported at
+        # the top of this module.
+        from gw2analytics_api.observability import shutdown_otel  # noqa: PLC0415
+
+        await asyncio.to_thread(shutdown_otel, timeout_s=5.0)
 
 
 app = FastAPI(