Skip to content

M3-21: bind real Active Metadata projection to S3-compatible warehouse #37

Description

@zhouning

Problem

M3-20 binds the real Chongqing Active Metadata projection to a restart-continuous Gravitino JDBC runtime, but its Iceberg warehouse is a local RWO PVC. M3-10 proves cross-node MinIO interoperability only for a synthetic Spark probe. Neither result proves that the real ResourceVersion promotion is bound to an object-store-backed provider runtime.

Goal

Create an independent object-store runtime-bound promotion candidate for the same immutable Chongqing ResourceVersion. Materialize its Gravitino projection in a JDBC catalog backed by a cross-node S3-compatible warehouse, inspect the Iceberg metadata directly, and require exact no-op replay before and after ordered PostgreSQL/Gravitino restart without changing M3-20 history.

Scope

  • Validate the full checked M3-20 and M3-10 dependency evidence before provider mutation.
  • Reuse the path-free Chongqing ResourceVersion and content SHA; do not commit source data or absolute paths.
  • Bind cluster, namespace, Gravitino Service, MinIO Service, PostgreSQL/Gravitino/MinIO StatefulSets, PostgreSQL/MinIO PVCs, images, JDBC URI, S3 warehouse, endpoint and bucket.
  • Require Gravitino and Spark to have no warehouse PVC.
  • Create through a schema-bounded Basic principal and deny catalog administration.
  • Require first apply to be one table mutation, immediate replay and first post-restart replay to be no_op/0.
  • Verify PostgreSQL and Gravitino pod rotation with stable provider/object-store identities.
  • Inspect the exact table prefix and Iceberg metadata through the S3 API before and after restart.
  • Emit sanitized evidence, ADR, Platform Truth/System-of-Record updates and CI validation.

Non-Goals

  • Changing or persisting M3-20/M3-19 binding or promotion history.
  • Claiming production object storage, durable catalog, protected identity, TLS/OIDC or production readiness.
  • Loading source feature rows or claiming production ingestion.
  • Running full Spark/Flink conformance.

Acceptance Criteria

  • Retained OpenMetadata and the new Gravitino table bind the same ResourceVersion UUID/content SHA.
  • The runtime binding includes stable JDBC and S3-compatible provider/storage identities and excludes rotating pod identities.
  • The Gravitino table creates Iceberg metadata under the expected S3 prefix; no file warehouse or Gravitino warehouse PVC is present.
  • Immediate and first post-restart apply are no_op/0 with identical logical binding and promotion candidate.
  • PostgreSQL and Gravitino pod UIDs rotate while MinIO, Service, StatefulSet and PVC identities remain stable.
  • Direct S3 table metadata remains identical across restart.
  • Runtime material, local paths and source data are absent from committed evidence; cleanup is complete.

Status

in_review

Pull Request

Execution Gate

allowed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions