Skip to content

[feature] spate-gcs: a Google Cloud Storage backfill source #751

Description

@MarcusKainth

What kind

A new connector (source, sink, or deserializer)

Which components

S3 source (spate-s3)

What you are trying to do, and what stops you

Part of #749. A backfill from Google Cloud Storage has no source: spate-s3 enables only the object_store aws feature (crates/spate-s3/Cargo.toml:27), so a gs:// URL fails when the store is built.

spate-gcs is a store crate over the backfill engine extracted in #750, with the gcp client built by the shared builder crate from #685. It carries:

  • GcsSource and GcsSourceConfig, with the same fields as S3SourceConfig and gs:// URLs;
  • the spate_gcs_source_* metric namespace;
  • a gcs feature on the spate facade;
  • a docs page under docs/user-guide/04-connectors/sources/.

Client hardening. spate-s3 builds its S3 client from the environment and applies hardened client defaults (crates/spate-s3/src/source.rs:403). The gcp builder also reads the environment, so the GCS client needs the same review of which variables it honours and which defaults it hardens, with tests.

Rejected credentials. spate-s3 recognises a listing rejected with 401 or 403 by matching the S3 client's error text (crates/spate-s3/src/error.rs:68), because object_store reports it as Generic. How the gcp client reports a rejected listing and a rejected read is unverified. A failing test against a local server pins both before the classification is written.

Split identity. Each store crate passes its own domain tag and fingerprint prefix to the engine, so a GCS job's persisted ids are never confused with an S3 job's.

This pull request adds the ADR for the one-crate-per-store layout, since it is the first time two store crates use the engine.

Blocked by #750 and by #685.

Line numbers are on a58ebf94.

Semver impact, as best you can tell

Additive — nothing existing changes

Does this touch any of these?

  • INV-1 — delivery semantics, or anything that could commit a watermark past unacknowledged data
  • INV-2 — the source poll loop, which must never block on a channel send
  • INV-3 — the checkpoint tracker, which is synchronous and free of async runtimes
  • INV-5 — the sink intake path, which must not await outside its select!
  • INV-6 — adding a connector or 0.x dependency type to a public signature in spate-core
  • INV-8, INV-9 or INV-10 — metric naming, cardinality, or gauge ownership

The crate registers a new spate_gcs_source_* namespace with the same families as spate_s3_source_*.

Would you want to implement it?

Yes, if the design is agreed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions