feat(controlplane): expose metadata Postgres proxy

CLAUDE.md

@@ -27,6 +27,69 @@ PG Client → TLS/Auth/PG Protocol → Control Plane pod
                                  → Flight SQL (TCP+TLS) → per-org Worker pod (DuckDB)
 ```
 
+### Native metadata Postgres proxy
+
+The Kubernetes control plane can expose explicitly opted-in CNPG-backed
+warehouse metadata databases on a separate SNI suffix
+(`DUCKGRES_METADATA_HOSTNAME_SUFFIXES`; managed-warehouse deployments use
+`.md.dev.postwh.com`, `.md.us.postwh.com`, or `.md.eu.postwh.com`). This is an
+early connection branch, not a DuckDB
+executor: the existing org `root` password authenticates at Duckgres, the
+startup database MUST be exactly `metadata`, and the control plane resolves
+the real endpoint/database/tenant role/password internally through
+`SharedWorkerActivator.MetadataPostgresURL`. After authenticating upstream
+with that internal credential, protocol traffic is relayed byte-for-byte. The
+endpoint and password remain internal; the upstream role and database can be
+visible to the fully privileged client through normal PostgreSQL introspection
+such as `current_user` and `current_database()`.
+
+Access is fail-closed on the warehouse row's `metadata_proxy_enabled` flag,
+`state=ready`, and `metadata_store_kind=cnpg-shard`. Never infer publication
+from shard placement and never pass a client-supplied upstream database,
+username, host, or password. `DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG`
+(default 20 per control-plane replica) bounds public sessions so they cannot
+exhaust the internal PgBouncer pool. These sessions participate in the
+existing per-user kill / disable fan-out, and an established session is closed
+if the warehouse gate stops being eligible. An admin warehouse PUT that
+explicitly includes `metadata_proxy_enabled` reloads the local config snapshot
+and notifies peer replicas; established sessions observe the new gate on their
+next five-second recheck after snapshot propagation.
+
+The initial scope is dedicated, single-customer CNPG shards only. Do not enable
+an org on a shared shard until upstream `CONNECT` ACLs and role hardening are in
+place. The exact virtual database check prevents selecting a different startup
+database, but after connection the customer `root` credential has the full
+access of the internally resolved metadata role.
+
+Observability stays explicit at the branch boundary:
+`duckgres_connections_open` includes these client sockets because it is the
+process-wide accepted-connection gauge, while
+`duckgres_metadata_proxy_connections_open`,
+`duckgres_metadata_proxy_connection_attempts_total`,
+`duckgres_metadata_proxy_connection_duration_seconds`,
+`duckgres_metadata_proxy_upstream_connect_duration_seconds`, and
+`duckgres_metadata_proxy_bytes_total` isolate proxy load and failures. The
+relay is intentionally opaque, so DuckDB query metrics, query logs, and query
+traces do not include SQL executed through it. Use the CNPG/PgBouncer metrics
+and the fixed upstream `application_name=duckgres-metadata-proxy` for
+database-side attribution; never make one org's metadata target part of the
+control-plane health check. Target resolution plus upstream
+connect/auth/synchronization has a fixed 10-second bootstrap deadline; the
+deadline is canceled after hijack and never applies to established relay
+traffic. `duckgres_auth_failures_total` includes wrong-password proxy attempts;
+the dedicated attempt counter with `outcome="auth_failed"` is the proxy split.
+Pre-TLS `duckgres_rate_limit_rejects_total` events cannot be assigned to either
+endpoint because SNI has not yet been observed.
+
+Metadata-proxy `CancelRequest` handling is session-terminating. Synthetic
+backend keys map to the exact established frontend/upstream connection pair on
+the owning control-plane replica, which closes both rather than redialing a
+PgBouncer Service where instance-local cancel keys could reach the wrong pod.
+Raw cancel connections remain control-plane-local behind the NLB, matching the
+existing cancellation locality: a synthetic-key miss on another replica is
+absorbed and counted by
+`duckgres_metadata_proxy_cancel_requests_total{outcome="not_local"}`.
+
 In topologies 2 and 3, the control plane also exposes an Arrow Flight SQL ingress (`--flight-port`) for clients that prefer Flight over the PG wire protocol.
 
 ### Key Components

README.md

@@ -68,12 +68,18 @@ labels, aggregation rules, PromQL examples, and admission metric migration.
 
 | Metric | Type | Description |
 |--------|------|-------------|
-| `duckgres_connections_open` | Gauge | Number of currently open client connections |
-| `duckgres_connection_duration_seconds{org}` | Histogram | Client connection lifetime, accept→disconnect (includes `_count`, `_sum`, `_bucket`); `_sum` is exact total connection-seconds (per org) with no scrape-integral bias. The disconnect log also carries `duration_ms` |
+| `duckgres_connections_open` | Gauge | Process-wide number of currently open client connections, including native metadata-proxy sockets |
+| `duckgres_connection_duration_seconds{org}` | Histogram | Worker-backed Duckgres connection lifetime, accept→disconnect (includes `_count`, `_sum`, `_bucket`); excludes native metadata-proxy connections, which use their dedicated duration family |
+| `duckgres_metadata_proxy_connections_open{org}` | Gauge | Current admitted native metadata Postgres proxy connections; process-local, so sum across control-plane replicas |
+| `duckgres_metadata_proxy_connection_attempts_total{org,outcome}` | Counter | Metadata proxy attempts by bounded terminal outcome |
+| `duckgres_metadata_proxy_connection_duration_seconds{org}` | Histogram | Lifetime of admitted metadata proxy connections, including upstream bootstrap |
+| `duckgres_metadata_proxy_upstream_connect_duration_seconds{org,outcome}` | Histogram | Internal metadata Postgres connect/auth latency; outcome is `success` or `error` |
+| `duckgres_metadata_proxy_bytes_total{org,direction}` | Counter | Post-authentication pgwire bytes relayed in `client_to_upstream` or `upstream_to_client` direction |
+| `duckgres_metadata_proxy_cancel_requests_total{outcome}` | Counter | Raw metadata-proxy CancelRequests handled as `session_terminated` on the owning control-plane replica or `not_local` on another replica |
 | `duckgres_query_total{org,status,reason}` | Counter | Total non-empty query attempts. Valid status/reason pairs: `success/none`; `failure/user`, `failure/canceled`, `failure/conflict`; `error/metadata_connection_lost`, `error/system`. |
 | `duckgres_query_duration_seconds{org}` | Histogram | Simple/extended query execution latency (includes `_count`, `_sum`, `_bucket`); use `duckgres_query_total` for attempt totals |
-| `duckgres_auth_failures_total` | Counter | Total number of authentication failures |
-| `duckgres_rate_limit_rejects_total` | Counter | Total number of connections rejected due to rate limiting |
+| `duckgres_auth_failures_total` | Counter | Process-wide authentication failures, including wrong-password metadata-proxy attempts; use `duckgres_metadata_proxy_connection_attempts_total{outcome="auth_failed"}` for the proxy-specific split |
+| `duckgres_rate_limit_rejects_total` | Counter | Process-wide pre-TLS connection rejections due to rate limiting; these cannot be attributed to the worker or metadata endpoint because SNI is not available yet |
 | `duckgres_rate_limited_ips` | Gauge | Number of currently rate-limited IP addresses |
 | `duckgres_flight_auth_sessions_active` | Gauge | Number of active Flight auth sessions on the control plane |
 | `duckgres_control_plane_workers_active` | Gauge | Number of active control-plane worker processes |
@@ -367,6 +373,8 @@ Run with config file:
 | `DUCKGRES_HANDOVER_DRAIN_TIMEOUT` | Max time to drain planned shutdowns and upgrades before forcing exit | `24h` in process mode, `15m` in remote K8s mode |
 | `DUCKGRES_SNI_ROUTING_MODE` | Multi-tenant managed-hostname routing: `off`, `passthrough`, or `enforce`. Postgres uses the requested dbname first; managed SNI must resolve to the same org, and SNI supplies the database only when dbname is empty. | `off` |
 | `DUCKGRES_MANAGED_HOSTNAME_SUFFIXES` | Comma-separated managed hostname suffixes such as `.dw.us.postwh.com` | - |
+| `DUCKGRES_METADATA_HOSTNAME_SUFFIXES` | Comma-separated SNI suffixes for the explicitly enabled native metadata Postgres proxy, such as `.md.dev.postwh.com`, `.md.us.postwh.com`, or `.md.eu.postwh.com` | - |
+| `DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG` | Maximum admitted metadata proxy sessions per org on each control-plane replica | `20` |
 | `DUCKGRES_DUCKLAKE_METADATA_STORE` | DuckLake metadata connection string | - |
 | `DUCKGRES_DUCKLAKE_DELTA_CATALOG_ENABLED` | Attach a Delta Lake catalog/table during worker boot/activation | `false` |
 | `DUCKGRES_DUCKLAKE_DELTA_CATALOG_PATH` | Delta Lake catalog/table path; defaults to sibling `delta/` prefix at the DuckLake object-store root when enabled | Derived |
@@ -889,6 +897,43 @@ In Kubernetes environments, `--worker-backend remote` is the multitenant path. I
 
 Managed-hostname routing is controlled by `--sni-routing-mode` and `--managed-hostname-suffixes`. For Postgres, an explicit startup `database`/`dbname` takes priority, but when SNI matches a managed suffix the hostname prefix and requested database must resolve to the same org. If the startup database is empty, the managed SNI prefix is used as the database fallback. Unknown `--sni-routing-mode` values behave like `off`.
 
+The native metadata Postgres proxy is a separate, fail-closed SNI path selected
+by `DUCKGRES_METADATA_HOSTNAME_SUFFIXES`. It is available only when an org's
+warehouse explicitly has `metadata_proxy_enabled=true`, is ready, and uses a
+CNPG-shard metadata store. The client must connect as `root` with the org's
+existing Duckgres password and must send the exact non-empty
+`dbname=metadata`. Duckgres resolves and uses the real metadata role, password,
+database, and PgBouncer endpoint internally. The endpoint and password are
+never sent to the client; the upstream role and database may be visible
+through normal PostgreSQL introspection such as `current_user` and
+`current_database()`. Managed-warehouse deployments configure their own
+environment suffix (`.md.dev.postwh.com`, `.md.us.postwh.com`, or
+`.md.eu.postwh.com`); suffixes are never inferred from the ordinary Duckgres
+hostname. The per-org connection limit is enforced independently on every
+control-plane replica. Internal target resolution and
+connect/auth/synchronization have a fixed 10-second bootstrap deadline; the
+deadline does not apply after the relay is established. An admin/UI update that
+includes `metadata_proxy_enabled` reloads the local config snapshot and
+notifies peer replicas; established sessions close on their next five-second
+authorization recheck after the updated snapshot arrives.
+
+The initial rollout is restricted operationally to dedicated,
+single-customer CNPG shards. Do not enable `metadata_proxy_enabled` for an org
+on a shared shard until upstream database `CONNECT` ACLs and role hardening are
+in place: once the exact `metadata` database connection is established, the
+customer's `root` credential intentionally receives full access available to
+the internally resolved metadata role.
+
+Metadata-proxy cancellation is deliberately session-terminating: when a raw
+PostgreSQL `CancelRequest` reaches the control-plane replica that owns the
+synthetic backend key, Duckgres closes that exact frontend and upstream
+connection pair. PgBouncer cancellation keys are instance-local, so Duckgres
+does not redial the pooler Service. Like existing Duckgres cancellation, the
+raw follow-up TCP connection is control-plane-local behind the NLB; a request
+routed to another replica is counted as
+`duckgres_metadata_proxy_cancel_requests_total{outcome="not_local"}` and cannot
+terminate the owning session.
+
 Workers are spawned on demand: when an org opens a session with no reusable worker, the control plane creates a worker pod (sized from the connection's `duckgres.worker_cpu`/`worker_memory` request, or a default), activates it over the worker control RPC, and it becomes hot for that org. When its last session ends, the worker moves to `hot_idle` instead of being retired immediately: it keeps the org assignment and DuckLake attachment so any control-plane replica can reclaim it for the same org (by exact worker shape) without full reactivation, until its `duckgres.worker_ttl` expires. Hot-idle reuse is image/version strict. The janitor retires hot-idle workers at their TTL, but `default_worker_min_hot_idle` lets an org retain a minimum number of compatible default-profile hot-idle workers by skipping TTL retirement when the count is already at or below the floor. The default is `0` (disabled). The main lifecycle is: idle → reserved → activating → hot → hot_idle → retired. Workers can also move through `draining` during shutdown, rollout, or cleanup. (Spawn latency is hidden by the node-headroom controller, which keeps placeholder pods ready for real workers to preempt.)
 
 ```bash

cmd/duckgres-controlplane/main.go

@@ -246,6 +246,8 @@ func main() {
 		ReadOnlySecretFallbacks:   resolved.ReadOnlySecretFallbacks,
 		SNIRoutingMode:             resolved.SNIRoutingMode,
 		ManagedHostnameSuffixes:    resolved.ManagedHostnameSuffixes,
+		MetadataHostnameSuffixes:   resolved.MetadataHostnameSuffixes,
+		MetadataProxyMaxConns:      resolved.MetadataProxyMaxConns,
 		DucklingBucketSuffix:       resolved.DucklingBucketSuffix,
 		DuckLakeDefaultSpecVersion: resolved.DuckLakeDefaultSpecVersion,
 

configresolve/resolve.go

@@ -133,6 +133,8 @@ type Resolved struct {
 	UserSecretKey                   string
 	SNIRoutingMode                  string
 	ManagedHostnameSuffixes         []string
+	MetadataHostnameSuffixes        []string
+	MetadataProxyMaxConns           int
 	DucklingBucketSuffix            string
 	DuckLakeDefaultSpecVersion      string
 
@@ -222,6 +224,8 @@ func ResolveEffective(fileCfg *configloader.FileConfig, cli CLIInputs, getenv fu
 	var userSecretKey string
 	var sniRoutingMode string
 	var managedHostnameSuffixes []string
+	var metadataHostnameSuffixes []string
+	metadataProxyMaxConnections := 20
 	var ducklingBucketSuffix string
 
 	if fileCfg != nil {
@@ -786,6 +790,18 @@ func ResolveEffective(fileCfg *configloader.FileConfig, cli CLIInputs, getenv fu
 	if v := getenv("DUCKGRES_MANAGED_HOSTNAME_SUFFIXES"); v != "" {
 		managedHostnameSuffixes = splitAndTrim(v, ",")
 	}
+	if v := getenv("DUCKGRES_METADATA_HOSTNAME_SUFFIXES"); v != "" {
+		metadataHostnameSuffixes = splitAndTrim(v, ",")
+	}
+	if v := getenv("DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG"); v != "" {
+		if n, err := strconv.Atoi(v); err == nil && n > 0 {
+			metadataProxyMaxConnections = n
+		} else if err != nil {
+			warn("Invalid DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG: " + err.Error())
+		} else {
+			warn("DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG must be > 0")
+		}
+	}
 	// Env-only (set by the duckgres Helm chart per environment). The env suffix
 	// the control plane uses to name a type=s3bucket Duckling's per-org bucket:
 	// posthog-duckling-<compact-org>-<suffix>. Must equal crossplane-config's
@@ -1244,6 +1260,8 @@ func ResolveEffective(fileCfg *configloader.FileConfig, cli CLIInputs, getenv fu
 		UserSecretKey:                   userSecretKey,
 		SNIRoutingMode:                  sniRoutingMode,
 		ManagedHostnameSuffixes:         managedHostnameSuffixes,
+		MetadataHostnameSuffixes:        metadataHostnameSuffixes,
+		MetadataProxyMaxConns:           metadataProxyMaxConnections,
 		DucklingBucketSuffix:            ducklingBucketSuffix,
 		DuckLakeDefaultSpecVersion:      cfg.DuckLake.SpecVersion,
 

configresolve/resolve_k8s_test.go

@@ -1,6 +1,7 @@
 package configresolve
 
 import (
+	"slices"
 	"testing"
 )
 
@@ -25,6 +26,26 @@ func TestResolveEffectiveExposesDuckLakeDefaultSpecVersionForControlPlane(t *tes
 	}
 }
 
+func TestResolveEffectiveParsesMetadataHostnameSuffixes(t *testing.T) {
+	resolved := ResolveEffective(nil, CLIInputs{}, func(key string) string {
+		switch key {
+		case "DUCKGRES_METADATA_HOSTNAME_SUFFIXES":
+			return " .md.dev.postwh.com, .md.us.postwh.com, .md.eu.postwh.com "
+		case "DUCKGRES_METADATA_PROXY_MAX_CONNECTIONS_PER_ORG":
+			return "7"
+		}
+		return ""
+	}, nil)
+
+	want := []string{".md.dev.postwh.com", ".md.us.postwh.com", ".md.eu.postwh.com"}
+	if !slices.Equal(resolved.MetadataHostnameSuffixes, want) {
+		t.Fatalf("expected metadata hostname suffixes %v, got %v", want, resolved.MetadataHostnameSuffixes)
+	}
+	if resolved.MetadataProxyMaxConns != 7 {
+		t.Fatalf("expected metadata per-org connection cap 7, got %d", resolved.MetadataProxyMaxConns)
+	}
+}
+
 func TestResolveEffectiveParsesK8sWorkerMaxTTL(t *testing.T) {
 	resolved := ResolveEffective(nil, CLIInputs{}, func(key string) string {
 		if key == "DUCKGRES_K8S_WORKER_MAX_TTL" {

controlplane/admin/api.go

@@ -747,6 +747,7 @@ func managedWarehouseUpsertColumns() []string {
 		// DB column name. Mismatching this against the actual column makes
 		// the ON CONFLICT … DO UPDATE clause throw 42703.
 		"duck_lake_version",
+		"metadata_proxy_enabled",
 		"duckling_name",
 		"warehouse_database_endpoint",
 		"warehouse_database_port",
@@ -807,6 +808,7 @@ type apiHandler struct {
 type managedWarehouseRequest struct {
 	Image                        string                                        `json:"image"`
 	DuckLakeVersion              string                                        `json:"ducklake_version"`
+	MetadataProxyEnabled         bool                                          `json:"metadata_proxy_enabled"`
 	DucklingName                 string                                        `json:"duckling_name"`
 	WarehouseDatabase            configstore.ManagedWarehouseDatabase          `json:"warehouse_database"`
 	MetadataStore                configstore.ManagedWarehouseMetadataStore     `json:"metadata_store"`
@@ -1420,8 +1422,16 @@ func (h *apiHandler) putManagedWarehouse(c *gin.Context) {
 	// NAMES only — the values can carry credentials/secret refs, so we never put
 	// them in the audit log). Gives "changed: metadata_store, s3" instead of a
 	// bare "warehouse.update".
-	if changed := topLevelJSONKeys(body); len(changed) > 0 {
-		setAuditDetail(c, "changed: "+strings.Join(changed, ", "))
+	changedFields := topLevelJSONKeys(body)
+	if len(changedFields) > 0 {
+		setAuditDetail(c, "changed: "+strings.Join(changedFields, ", "))
+	}
+	metadataProxyChanged := false
+	for _, field := range changedFields {
+		if field == "metadata_proxy_enabled" {
+			metadataProxyChanged = true
+			break
+		}
 	}
 
 	// MutateManagedWarehouse locks the row inside a transaction, runs the
@@ -1467,6 +1477,16 @@ func (h *apiHandler) putManagedWarehouse(c *gin.Context) {
 		c.JSON(http.StatusNotFound, gin.H{"error": "org not found"})
 		return
 	}
+	if metadataProxyChanged {
+		// The proxy gate is consumed from each CP's in-memory snapshot. Reload
+		// locally and notify peers now so an explicit admin/UI toggle does not
+		// wait for the normal config poll interval. Established sessions
+		// recheck the refreshed gate at most five seconds later.
+		if err := h.notifyPeersOfChange(c); err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+	}
 	c.JSON(http.StatusOK, stored)
 }