What kind
A new connector (source, sink, or deserializer)
Which components
S3 source (spate-s3)
What you are trying to do, and what stops you
Part of #749. A backfill from Azure Blob Storage has no source: spate-s3 enables only the object_store aws feature (crates/spate-s3/Cargo.toml:27), so an az:// URL fails when the store is built.
spate-azure is a store crate over the backfill engine extracted in #750, with the azure client built by the shared builder crate from #685. It carries:
AzureSource and AzureSourceConfig, with the same fields as S3SourceConfig and az:// URLs;
- the
spate_azure_source_* metric namespace;
- an
azure feature on the spate facade;
- a docs page under
docs/user-guide/04-connectors/sources/.
Client hardening. spate-s3 builds its S3 client from the environment and applies hardened client defaults (crates/spate-s3/src/source.rs:403). The azure builder also reads the environment, so the Azure client needs the same review of which variables it honours and which defaults it hardens, with tests.
Rejected credentials. spate-s3 recognises a listing rejected with 401 or 403 by matching the S3 client's error text (crates/spate-s3/src/error.rs:68), because object_store reports it as Generic. How the azure client reports a rejected listing and a rejected read is unverified. A failing test against a local server pins both before the classification is written.
Split identity. Each store crate passes its own domain tag and fingerprint prefix to the engine, so an Azure job's persisted ids are never confused with an S3 job's.
Blocked by #750 and by #685. Lands after #751, which adds the ADR for the crate layout.
Line numbers are on a58ebf94.
Semver impact, as best you can tell
Additive — nothing existing changes
Does this touch any of these?
The crate registers a new spate_azure_source_* namespace with the same families as spate_s3_source_*.
Would you want to implement it?
Yes, if the design is agreed
What kind
A new connector (source, sink, or deserializer)
Which components
S3 source (spate-s3)
What you are trying to do, and what stops you
Part of #749. A backfill from Azure Blob Storage has no source:
spate-s3enables only theobject_storeawsfeature (crates/spate-s3/Cargo.toml:27), so anaz://URL fails when the store is built.spate-azureis a store crate over the backfill engine extracted in #750, with theazureclient built by the shared builder crate from #685. It carries:AzureSourceandAzureSourceConfig, with the same fields asS3SourceConfigandaz://URLs;spate_azure_source_*metric namespace;azurefeature on thespatefacade;docs/user-guide/04-connectors/sources/.Client hardening.
spate-s3builds its S3 client from the environment and applies hardened client defaults (crates/spate-s3/src/source.rs:403). Theazurebuilder also reads the environment, so the Azure client needs the same review of which variables it honours and which defaults it hardens, with tests.Rejected credentials.
spate-s3recognises a listing rejected with 401 or 403 by matching the S3 client's error text (crates/spate-s3/src/error.rs:68), becauseobject_storereports it asGeneric. How theazureclient reports a rejected listing and a rejected read is unverified. A failing test against a local server pins both before the classification is written.Split identity. Each store crate passes its own domain tag and fingerprint prefix to the engine, so an Azure job's persisted ids are never confused with an S3 job's.
Blocked by #750 and by #685. Lands after #751, which adds the ADR for the crate layout.
Line numbers are on
a58ebf94.Semver impact, as best you can tell
Additive — nothing existing changes
Does this touch any of these?
select!spate-coreThe crate registers a new
spate_azure_source_*namespace with the same families asspate_s3_source_*.Would you want to implement it?
Yes, if the design is agreed