diff --git a/workload-generator/README.md b/workload-generator/README.md index 9e50f31b..ba38a05c 100644 --- a/workload-generator/README.md +++ b/workload-generator/README.md @@ -8,7 +8,7 @@ Write a script that creates Delta tables with normal SQL, declare what specs to | Document | Description | |----------|-------------| -| **[Spec Format Reference](docs/spec-reference.md)** | JSON schema for the `read` and `snapshot` spec types, with examples | +| **[Spec Format Reference](docs/spec-reference.md)** | JSON schema for the `read`, `snapshot`, and `write` spec types, with examples | | **[Harness Implementation Guide](docs/harness-implementation-guide.md)** | Step-by-step guide to build a test harness that runs workloads against your engine, with a Rust example | | **[Authoring Guide](docs/authoring-guide.md)** | How to write new workload suites, patterns, recipes, and debugging tips | @@ -81,8 +81,10 @@ WORKLOAD_FORCE=true WORKLOAD_OUTPUT_DIR=/tmp/workloads sbt "testOnly *ReadsSuite | Component | Purpose | |-----------|---------| | `WorkloadTestSuite` | ScalaTest base class with workload generation integration | -| `WorkloadOps` | DSL trait: `sql()`, `registerTable()`, `readSpec()`, `snapshotSpec()` | -| `WorkloadGenerator` | Orchestrates table copy, spec capture, and validation | +| `WorkloadOps` | DSL trait (pass-through over `WorkloadContext`): `sql()`, `registerTable()`, `readSpec()`, write ops | +| `WorkloadContext` | DSL state: tracks created tables, declared specs, and write builders | +| `WorkloadGenerator` | Orchestrates table copy, spec capture, and self-validation | +| `WorkloadValidator` | Replays/validates a generated workload tree; presubmit acceptance entry point | | `ReadCapture` | Captures read specs with expected row data | | `SnapshotCapture` | Captures snapshot specs with protocol/metadata | | `TableInfoWriter` | Writes table metadata (schema, protocol, stats) | @@ -157,6 +159,9 @@ See the [Spec Format Reference](docs/spec-reference.md) for the complete JSON sc |------|--------------|---------| | **Read** | Data reads with time travel, predicates, column projection, data skipping | [Reference](docs/spec-reference.md#read-spec) | | **Snapshot** | Protocol and metadata reconstruction from log replay | [Reference](docs/spec-reference.md#snapshot-spec) | +| **Write** | Table construction via a portable sequence of commits (create/replace/insert/delete/update/schema-evolution/properties and low-level `commit`s), replayed into a fresh table | [Reference](docs/spec-reference.md#write-spec) | + +The write DSL (`createTableOp`, `insertOp`, `commitOp`, …) is exposed by `WorkloadOps`; see the [Authoring Guide](docs/authoring-guide.md) for how to author write workloads. A `read`/`snapshot` spec captured against a write-built table carries a `writeSpec` pointer and is validated *portably* (replay the sibling write spec into a fresh table, then compare). ## Workload Suites diff --git a/workload-generator/docs/authoring-guide.md b/workload-generator/docs/authoring-guide.md index dab09d1e..1b35a0bc 100644 --- a/workload-generator/docs/authoring-guide.md +++ b/workload-generator/docs/authoring-guide.md @@ -92,9 +92,26 @@ Inside a test body, these methods are available directly (via `WorkloadOps` trai | SQL | `sql(statement)` | — | | Table handles | `registerTable(name)`, `registerTableFromPath(path)` | → `TableHandle` | | Read specs | `readSpec(t, ...)`, `snapshotSpec(t, ...)` | `TableHandle` → `SpecRef` | +| Write specs | `createTableOp`, `replaceTableOp`, `insertOp`, `deleteOp`, `updateOp`, `addColumnsOp`, `renameColumnOp`, `dropColumnsOp`, `setPropertiesOp`, `unsetPropertiesOp`, `commitOp` (low-level) | → `WriteHandle`; `registerWriteSpec(w)` → `TableHandle` | | Checkpointing | `forceCheckpoint(tableName)` — triggers a checkpoint via DeltaLog | — | | Mutations | `mutateTable(t) { dir => ... }`, `modifyCommitActions(t, version) { ... }` | `TableHandle` | +### Authoring a write workload + +A write workload builds a table through DSL ops instead of raw `sql(...)`, producing a portable `write` spec (`specs/_write.json`) plus any `read`/`snapshot` specs derived from the built table. Start with `createTableOp` (or `replaceTableOp`), apply ops, then `registerWriteSpec(w)` to get a `TableHandle` you can attach reads/snapshots to: + +```scala +val w = createTableOp("tbl", schema = "id INT, name STRING") +insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) +// Low-level commit: logical rows are written through the engine (stats/partitionValues derived). +commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) +val t = registerWriteSpec(w) +readSpec(t, name = "read_all") +snapshotSpec(t) +``` + +`commitOp` returns the commit's ordinal (== table version); a later `removeFiles = Some(Seq(ordinal))` tombstones the files that commit added. See `WriteBasicSuite`/`WriteCommitSuite`/`WriteSequencesSuite` for worked examples and the [Write Spec reference](spec-reference.md#write-spec) for the on-disk format. + --- ## Creating Tables @@ -311,23 +328,22 @@ test("missing_file", "Test missing data file") { ### modifyCommitActions — Action-Level Manipulation -The modifier receives the full list of `(actionType, innerNode)` pairs for a commit and returns the (possibly reordered, filtered, or modified) list to write back. +The modifier receives the commit's typed `Seq[Action]` (subtypes `AddFile`, `RemoveFile`, `Metadata`, `Protocol`, `Txn`, `DomainMetadata`, …; unrecognized or malformed lines arrive as `RawAction` and pass through unchanged) and returns the reordered, filtered, or modified list to write back. Use `.copy` to edit fields. For byte-exact corruption, use the raw `mutateTable` path below instead. ```scala +import io.delta.workload.log.{AddFile, Metadata} + // Modify add actions -modifyCommitActions(t, version = 1) { actions => - actions.map { case ("add", node) => - node.put("stats", """{"numRecords":999}"""); ("add", node) - case other => other - } +modifyCommitActions(t, version = 1) { + _.map { case a: AddFile => a.copy(stats = Some("""{"numRecords":999}""")); case other => other } } // Drop all add actions -modifyCommitActions(t, version = 1) { _.filter(_._1 != "add") } +modifyCommitActions(t, version = 1) { _.filterNot(_.isInstanceOf[AddFile]) } -// Reorder — put metaData last +// Reorder: put metaData last modifyCommitActions(t, version = 0) { actions => - val (meta, rest) = actions.partition(_._1 == "metaData") + val (meta, rest) = actions.partition(_.isInstanceOf[Metadata]) rest ++ meta } ``` diff --git a/workload-generator/docs/harness-implementation-guide.md b/workload-generator/docs/harness-implementation-guide.md index a24ac1b2..f04bde52 100644 --- a/workload-generator/docs/harness-implementation-guide.md +++ b/workload-generator/docs/harness-implementation-guide.md @@ -207,6 +207,12 @@ Any spec type can have `"expectedError"` instead of `"expected"`: Run the operation and assert it fails. Matching the exact error code is ideal but optional — just asserting failure is a valid starting point. +### Write Specs + +A `write` spec (`specs/_write.json`) is a portable recipe for *building* a table: an ordered list of `commits` your harness replays into a fresh table, then compares against `expected/_write/`. High-level commits (create/replace/insert/delete/update/schema-evolution/properties) are replayed through your engine's normal write APIs; low-level `commit`s carry raw actions (the data files hold logical rows your engine writes through its own write path, so physical names/stats are engine-derived, and removes reference a prior add by commit ordinal == table version). See the [Write Spec reference](spec-reference.md#write-spec) for the field-level contract. + +Because a write spec is replayed (not byte-compared), comparison is **portable**: rows-only data checks, capability-based protocol comparison (your table may use explicit table features where Spark used legacy versions), and column-mapping-normalized schema. A `read`/`snapshot` spec carrying a `writeSpec` pointer is *write-derived* — reconstruct the table by replaying that sibling write spec first, then validate the read/snapshot against it (portably), rather than against a captured `delta/` directory. + --- ## Step 4: Incremental Adoption diff --git a/workload-generator/docs/spec-reference.md b/workload-generator/docs/spec-reference.md index 28c68014..02b6628f 100644 --- a/workload-generator/docs/spec-reference.md +++ b/workload-generator/docs/spec-reference.md @@ -22,6 +22,7 @@ Exactly one of `expected` or `expectedError` is present. The other is omitted (n - [Common Types](#common-types) - [Read Spec](#read-spec) - [Snapshot Spec](#snapshot-spec) +- [Write Spec](#write-spec) - [table_info.json](#table_infojson) - [Expected Data Layout](#expected-data-layout) @@ -80,6 +81,7 @@ Tests reading data from a Delta table with optional time travel, predicate pushd | Field | Type | Required | Description | |-------|------|----------|-------------| | `type` | `string` | yes | Always `"read"` | +| `writeSpec` | `string` | no | If present, this read is *write-derived*: the named sibling write spec (`specs/`) must be replayed into a fresh table first, then this read validated against it. Absent ⇒ read-only (validate against the captured table). | | `version` | `long` | no | Time travel to this version | | `timestamp` | `string` | no | Time travel to this timestamp (format: `yyyy-MM-dd HH:mm:ss.SSS`) | | `predicate` | `string` | no | SQL WHERE clause to apply (e.g., `"id > 5"`) | @@ -243,6 +245,7 @@ Tests constructing a table snapshot at a given version, validating the protocol | Field | Type | Required | Description | |-------|------|----------|-------------| | `type` | `string` | yes | Always `"snapshot"` | +| `writeSpec` | `string` | no | If present, write-derived: replay the named sibling write spec first, then validate this snapshot against it (portable comparison). Absent ⇒ validate against the captured table. | | `version` | `long` | no | Snapshot at this version (latest if omitted and no timestamp) | | `timestamp` | `string` | no | Snapshot at this timestamp | | `expected` | `SnapshotExpected` | no | Present on success | @@ -347,6 +350,209 @@ These are the raw Delta protocol and metadata JSON structures — not simplified --- +## Write Spec + +**Type:** `"write"` + +Tests Delta *writer* implementations: a portable, engine-agnostic recipe of `commits` to replay into a fresh table, after which the result is compared against `expected/_write/`. + +It is **just another spec** — it lives at `specs/_write.json` and dispatches by `type`. The read/snapshot specs of the same workload carry a `writeSpec` pointer back to it, marking them *write-derived* (replayed from the write spec, then validated against it). + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `type` | `string` | yes | Always `"write"` | +| `commits` | `WriteCommit[]` | yes | Ordered list of commits to replay | + +### WriteCommit + +Each commit represents a single Delta transaction. There are two categories: high-level operations (SQL semantics) and the low-level `commit` (raw Delta actions). + +#### High-Level Operations + +These map to SQL-like operations. The writer translates them to appropriate Delta actions. + +| Operation | Fields | Description | +|-----------|--------|-------------| +| `create_table` | `schema`, `partitionColumns?`, `properties?` | Create a new table with the given schema (`schema` is a Delta-JSON struct) | +| `replace_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Replace the table's schema/partitioning/properties (and all data). With `dataFiles` it is a replace-as-select: a single commit that also writes the bundled data. | +| `insert` | `dataFiles?` | Append the rows in the bundled Parquet data files | +| `update` | `predicate`, `set` | Update rows matching `predicate`; `set` maps column → SQL expression | +| `delete` | `predicate` | Delete rows matching `predicate` (SQL WHERE clause) | +| `evolve_schema` | `addColumns?`, `renameColumns?`, `dropColumns?` | Modify table schema. `addColumns` entries are `{name, type, nullable}`; `renameColumns` maps old → new | +| `update_properties` | `set?`, `remove?` | Modify table properties (`set` map, `remove` names) | + +The data for `insert` and `replace_table` is bundled as Parquet under `data/commit_N/` (the +generator's authoring API accepts in-memory rows, but the spec always stores Parquet). A +consumer replays by reading those files — `insert` appends them; `replace_table` performs a +single replace-as-select from them. + +`RENAME COLUMN` / `DROP COLUMN` in `evolve_schema` require the table to have column mapping +enabled (a Delta/Spark constraint). + +#### Low-Level Operation + +| Operation | Fields | Description | +|-----------|--------|-------------| +| `commit` | `schema?`, `tableProperties?`, `txn?`, `addFiles?`, `removeFiles?`, `addDomainMetadata?`, `removeDomainMetadata?` | Bundle raw Delta actions in one commit. Each `addFiles[]` entry's logical rows are bundled as Parquet (`dataFile`); a consumer writes them **through its own engine write path** (so column mapping, partitioning, and stats are handled — physical names/stats are not in the spec) and commits the resulting `AddFile`s alongside the `txn`/`DomainMetadata`/schema/property changes. `removeFiles[]` tombstone a file added by a prior low-level `commit`, referenced by that commit's ordinal (`addedAtCommit`); since the engine assigns paths per table, the consumer resolves the ordinal to its own table's path. Deletion vectors are out of scope. | + +### Low-level value types + +Value types referenced by the low-level `commit` operation. An `AddFileAction` carries a pointer to a Parquet of **logical** rows (full rows incl. partition columns); the consumer writes it through its own engine write path, so physical names, partitioning, and stats are derived per table, not stored. + +| Type | Field | Description | +|------|-------|-------------| +| `AddFileAction` | `dataFile` | Relative path under `data/commit_N/` to the Parquet of logical rows | +| `AddFileAction` | `dataChange?` | Whether this is a data change (default true) | +| `RemoveFileAction` | `addedAtCommit` | Ordinal (== table version) of the prior low-level `commit` that added the file(s) to tombstone; consumer resolves it to its own table's path(s) | +| `RemoveFileAction` | `dataChange?` | Whether this is a data change (default true) | +| `AppTxn` | `appId`, `version` | Application id and its monotonic transaction version for idempotent writes (`txn` action) | +| `AddDomainMetadata` | `domain`, `configuration` | Domain name and its configuration payload | + +### Expected Data + +The final table state after replaying all commits is captured under `expected/_write/`: + +- `table_content/` — Parquet files with the expected rows +- `table_version_metadata.json` — the latest snapshot's `protocol` and `metadata` + +### Comparison Semantics + +The capture comes from one writer, so a consumer compares the replayed table against `expected` *portably*, not byte-for-byte: + +- **rows** (`table_content/`) — multiset equality (order-independent). +- **protocol** — the replay must support at least the expected reader/writer versions and features (a stronger protocol is acceptable). +- **configuration** — only the keys the spec's own commits declared: `create_table`/`replace_table` `properties` + `update_properties.set` minus `.remove` (`replace_table` resets the set); each present and equal, removed keys absent. Engine-injected defaults are ignored. +- **schemaString** — equal with per-field column-mapping `physicalName`/`id` normalized out (minted per table); `partitionColumns` and `format` equal. + +### Directory Structure + +``` +/ +├── delta/ # the captured table (for read-only validation) +├── specs/ +│ ├── _read*.json +│ ├── _snapshot.json +│ └── _write.json # the write spec — just another spec +├── data/ # Parquet referenced by write commits +│ ├── commit_1/ +│ │ └── part-00000.parquet +│ └── ... +├── expected/ +│ ├── _write/ # write spec's final state +│ │ ├── table_content/ +│ │ └── table_version_metadata.json +│ └── / # expected_data for each read spec +│ └── expected_data/ +└── table_info.json +``` + +### Examples + +**High-level sequence (create, insert, delete, update):** + +```json +{ + "type": "write", + "commits": [ + { + "operation": "create_table", + "schema": { + "type": "struct", + "fields": [ + { "name": "id", "type": "integer", "nullable": false, "metadata": {} }, + { "name": "status", "type": "string", "nullable": true, "metadata": {} } + ] + }, + "properties": { "delta.enableDeletionVectors": "true" } + }, + { "operation": "insert", "dataFiles": ["data/commit_1/part-0000-abc.parquet"] }, + { "operation": "delete", "predicate": "id > 100" }, + { + "operation": "update", + "predicate": "status = 'pending'", + "set": { "status": "'active'" } + } + ] +} +``` + +**Schema evolution (add column):** + +```json +{ + "type": "write", + "commits": [ + { + "operation": "create_table", + "schema": { + "type": "struct", + "fields": [ + { "name": "id", "type": "integer", "nullable": false, "metadata": {} } + ] + } + }, + { "operation": "insert", "dataFiles": ["data/commit_1/part-0000-abc.parquet"] }, + { + "operation": "evolve_schema", + "addColumns": [ { "name": "email", "type": "string", "nullable": true } ] + }, + { "operation": "insert", "dataFiles": ["data/commit_3/part-0000-def.parquet"] } + ] +} +``` + +**Replace-as-select (replace schema + data in one commit):** + +```json +{ + "type": "write", + "commits": [ + { "operation": "create_table", "schema": { } }, + { "operation": "insert", "dataFiles": ["data/commit_1/part-00000.parquet"] }, + { + "operation": "replace_table", + "schema": { + "type": "struct", + "fields": [ + { "name": "id", "type": "integer", "nullable": true, "metadata": {} }, + { "name": "label", "type": "string", "nullable": true, "metadata": {} } + ] + }, + "dataFiles": ["data/commit_2/part-00000.parquet"] + } + ] +} +``` + +**Low-level commit (addFiles/removeFiles + txn + domainMetadata):** + +```json +{ + "type": "write", + "commits": [ + { + "operation": "create_table", + "schema": { }, + "properties": { "delta.feature.domainMetadata": "supported" } + }, + { "operation": "commit", "addFiles": [ { "dataFile": "data/commit_1/add_0.parquet" } ] }, + { + "operation": "commit", + "txn": { "appId": "streaming-job-1", "version": 42 }, + "addFiles": [ { "dataFile": "data/commit_2/add_0.parquet" } ], + "removeFiles": [ { "addedAtCommit": 1 } ], + "addDomainMetadata": [ + { "domain": "myApp.config", "configuration": "{\"version\": 1}" } + ] + } + ] +} +``` + +--- + ## table_info.json Written to the workload output root. Provides metadata about the generated table for discovery and filtering. diff --git a/workload-generator/src/main/scala/io/delta/workload/JsonUtil.scala b/workload-generator/src/main/scala/io/delta/workload/JsonUtil.scala index cfb48003..1ec0a1ae 100644 --- a/workload-generator/src/main/scala/io/delta/workload/JsonUtil.scala +++ b/workload-generator/src/main/scala/io/delta/workload/JsonUtil.scala @@ -28,6 +28,7 @@ import org.apache.spark.sql.{Column, DataFrame, SparkSession} import org.apache.spark.sql.functions._ import io.delta.workload.deltaharness.{LogView, SnapshotView} +import io.delta.workload.log.{AddFile, CommitLog} // ============================================================================= // Spec Expected types - success data or error info @@ -49,9 +50,11 @@ case class SnapshotExpected(protocol: Any, metadata: Any) // Spec case classes // ============================================================================= -@JsonPropertyOrder(Array("type", "version", "timestamp", "predicate", "columns", "expected", "expectedError")) +@JsonPropertyOrder(Array("type", "writeSpec", "version", "timestamp", "predicate", "columns", + "expected", "expectedError")) @JsonInclude(JsonInclude.Include.NON_ABSENT) case class ReadSpec( + writeSpec: Option[String] = None, version: Option[Long] = None, timestamp: Option[String] = None, predicate: Option[String] = None, @@ -61,9 +64,10 @@ case class ReadSpec( val `type`: String = "read" } -@JsonPropertyOrder(Array("type", "version", "timestamp", "expected", "expectedError")) +@JsonPropertyOrder(Array("type", "writeSpec", "version", "timestamp", "expected", "expectedError")) @JsonInclude(JsonInclude.Include.NON_ABSENT) case class SnapshotSpec( + writeSpec: Option[String] = None, version: Option[Long] = None, timestamp: Option[String] = None, expected: Option[SnapshotExpected] = None, @@ -118,32 +122,176 @@ case class TableInfo( // Low-level action types // ============================================================================= -/** Deletion vector descriptor for low-level commits. */ -@JsonPropertyOrder(Array("storageType", "pathOrInlineDv", "offset", "sizeInBytes", "cardinality")) -@JsonInclude(JsonInclude.Include.NON_ABSENT) -case class DeletionVectorAction( - storageType: String, - pathOrInlineDv: String, - offset: Option[Int] = None, - sizeInBytes: Int, - cardinality: Long) - -/** Add file action for low-level commits. */ -@JsonPropertyOrder(Array("dataFile", "partitionValues", "dataChange", "deletionVector")) +/** + * Add-file action serialized in a low-level `commit`. `dataFile` points to a Parquet of LOGICAL + * rows under `data/commit_N/`; on replay the engine writes it into the table (column-mapping- and + * partition-aware, stats computed), so physical names/stats are derived per table, never stored. + */ +@JsonPropertyOrder(Array("dataFile", "dataChange")) @JsonInclude(JsonInclude.Include.NON_ABSENT) case class AddFileAction( dataFile: String, - partitionValues: Option[Map[String, String]] = None, - dataChange: Option[Boolean] = None, - deletionVector: Option[DeletionVectorAction] = None) + dataChange: Option[Boolean] = None) -/** Remove file action for low-level commits. */ -@JsonPropertyOrder(Array("path", "dataChange")) +/** + * Remove-file action (tombstone) for low-level commits. References a prior low-level add by the + * commit ordinal that produced it (`addedAtCommit`); the engine assigns file paths per table, so + * the tombstone is resolved to the actual path(s) at replay. The tombstone inherits the live + * add's partitionValues/size/stats (extendedFileMetadata). + */ +@JsonPropertyOrder(Array("addedAtCommit", "dataChange")) @JsonInclude(JsonInclude.Include.NON_ABSENT) case class RemoveFileAction( - path: String, + addedAtCommit: Int, dataChange: Option[Boolean] = None) +/** Logical `rows` for a low-level add (full schema incl. partition columns); engine-materialized, not serialized. */ +case class AddFileInput( + rows: Seq[Map[String, Any]], + dataChange: Option[Boolean] = None) + +/** Application transaction for idempotent low-level commits. */ +@JsonPropertyOrder(Array("appId", "version")) +case class AppTxn(appId: String, version: Long) + +/** Domain metadata entry for low-level commits (added domains). */ +@JsonPropertyOrder(Array("domain", "configuration")) +case class AddDomainMetadata( + domain: String, + configuration: String) + +// ============================================================================= +// Write spec case classes +// ============================================================================= + +/** + * A single commit in a write spec. Each operation is its own case class carrying only the + * fields it uses, so illegal field combinations are unrepresentable and `match`es over the + * hierarchy are exhaustive. The `operation` discriminator is written by Jackson as the leading + * JSON property (`@JsonTypeInfo`); the in-memory `operation` accessor is `@JsonIgnore`d so it is + * not also serialized as a duplicate field. + * + * High-level ops (create_table/replace_table/insert/update/delete/evolve_schema/ + * update_properties) replay from their parameters; the low-level [[LowLevelCommitOp]] replays its + * recorded raw actions. + */ +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY, + property = "operation") +@JsonSubTypes(Array( + new JsonSubTypes.Type(value = classOf[CreateTableCommit], name = "create_table"), + new JsonSubTypes.Type(value = classOf[ReplaceTableCommit], name = "replace_table"), + new JsonSubTypes.Type(value = classOf[InsertCommit], name = "insert"), + new JsonSubTypes.Type(value = classOf[DeleteCommit], name = "delete"), + new JsonSubTypes.Type(value = classOf[UpdateCommit], name = "update"), + new JsonSubTypes.Type(value = classOf[EvolveSchemaCommit], name = "evolve_schema"), + new JsonSubTypes.Type(value = classOf[UpdatePropertiesCommit], name = "update_properties"), + new JsonSubTypes.Type(value = classOf[LowLevelCommitOp], name = "commit"))) +sealed trait WriteCommit { + @JsonIgnore def operation: String +} + +// ---- High-level operations (replayed from parameters) ---- + +@JsonPropertyOrder(Array("schema", "partitionColumns", "properties")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class CreateTableCommit( + schema: Any, + partitionColumns: Option[Seq[String]] = None, + properties: Option[Map[String, String]] = None) extends WriteCommit { + @JsonIgnore val operation = "create_table" +} + +/** + * `CREATE OR REPLACE TABLE` — replaces schema/partitioning/properties and all data. With `rows` + * it is a replace-as-select (RTAS): a single commit that replaces the table and writes the rows + * (replayed via `CREATE OR REPLACE TABLE … USING delta … AS SELECT … FROM VALUES`). + */ +@JsonPropertyOrder(Array("schema", "partitionColumns", "properties", "dataFiles")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class ReplaceTableCommit( + schema: Any, + partitionColumns: Option[Seq[String]] = None, + properties: Option[Map[String, String]] = None, + dataFiles: Option[Seq[String]] = None) extends WriteCommit { + @JsonIgnore val operation = "replace_table" +} + +@JsonPropertyOrder(Array("dataFiles")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class InsertCommit(dataFiles: Option[Seq[String]] = None) extends WriteCommit { + @JsonIgnore val operation = "insert" +} + +@JsonPropertyOrder(Array("predicate")) +case class DeleteCommit(predicate: String) extends WriteCommit { + @JsonIgnore val operation = "delete" +} + +@JsonPropertyOrder(Array("predicate", "set")) +case class UpdateCommit(predicate: String, set: Map[String, String]) extends WriteCommit { + @JsonIgnore val operation = "update" +} + +@JsonPropertyOrder(Array("addColumns", "renameColumns", "dropColumns")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class EvolveSchemaCommit( + addColumns: Option[Any] = None, + renameColumns: Option[Map[String, String]] = None, + dropColumns: Option[Seq[String]] = None) extends WriteCommit { + @JsonIgnore val operation = "evolve_schema" +} + +@JsonPropertyOrder(Array("set", "remove")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class UpdatePropertiesCommit( + set: Option[Map[String, String]] = None, + remove: Option[Seq[String]] = None) extends WriteCommit { + @JsonIgnore val operation = "update_properties" +} + +// ---- Low-level operation (raw Delta actions; data written via the engine on replay) ---- + +@JsonPropertyOrder(Array("schema", "tableProperties", "txn", "addFiles", "removeFiles", + "addDomainMetadata", "removeDomainMetadata")) +@JsonInclude(JsonInclude.Include.NON_ABSENT) +case class LowLevelCommitOp( + schema: Option[Any] = None, + tableProperties: Option[Map[String, String]] = None, + txn: Option[AppTxn] = None, + addFiles: Option[Seq[AddFileAction]] = None, + removeFiles: Option[Seq[RemoveFileAction]] = None, + addDomainMetadata: Option[Seq[AddDomainMetadata]] = None, + removeDomainMetadata: Option[Seq[String]] = None) extends WriteCommit { + @JsonIgnore val operation = "commit" +} + +@JsonPropertyOrder(Array("type", "commits")) +case class WriteSpec(commits: Seq[WriteCommit]) { + val `type`: String = "write" +} + +/** + * Layout conventions for a workload output directory, shared by capture and replay so the + * "commit index -> data directory" decision lives in one place (not duplicated across + * [[WriteSpecBuilder]] and [[WorkloadValidator]]). + */ +private[workload] object SpecLayout { + /** Relative path (from the output dir) of the data directory for commit `idx`. */ + def commitDataDir(idx: Int): String = s"data/commit_$idx" + + /** Relative path (from the output dir) of `name` under commit `idx`'s data directory. */ + def commitDataFile(idx: Int, name: String): String = s"${commitDataDir(idx)}/$name" + + /** + * The in-table `AddFile.path`s a commit produced, read from its `_delta_log/.json`. + * Because commit index == table version, a low-level remove's `addedAtCommit` ordinal is the + * version whose adds it tombstones — resolved here against the actual (engine-assigned) paths. + */ + def addPathsAt(tablePath: Path, version: Int): Seq[String] = + if (!Files.exists(CommitLog.commitFile(tablePath, version))) Seq.empty + else CommitLog.read(tablePath, version).collect { case a: AddFile => a.path } +} + // ============================================================================= // Delta log action case classes (for parsing commit JSON) // ============================================================================= @@ -228,6 +376,9 @@ object JsonUtil { def readSnapshotSpec(path: Path): SnapshotSpec = mapper.readValue(Files.readAllBytes(path), classOf[SnapshotSpec]) + def readWriteSpec(path: Path): WriteSpec = + mapper.readValue(Files.readAllBytes(path), classOf[WriteSpec]) + def toRowMultiset(df: DataFrame): Map[String, Int] = { // Drop MAP-type columns that Spark can't serialize to JSON val nonMapCols = df.schema.fields diff --git a/workload-generator/src/main/scala/io/delta/workload/ReadCapture.scala b/workload-generator/src/main/scala/io/delta/workload/ReadCapture.scala index 90cb999a..c6dd7cec 100644 --- a/workload-generator/src/main/scala/io/delta/workload/ReadCapture.scala +++ b/workload-generator/src/main/scala/io/delta/workload/ReadCapture.scala @@ -32,7 +32,7 @@ object ReadCapture { outputDir: Path, specsDir: Path, name: String, predicate: Option[String] = None, version: Option[Long] = None, timestamp: Option[String] = None, columns: Option[Seq[String]] = None, - expectError: Option[String] = None): Unit = { + expectError: Option[String] = None, writeSpec: Option[String] = None): Unit = { val specName = s"${testId}_$name" require(!(version.isDefined && timestamp.isDefined), @@ -44,7 +44,7 @@ object ReadCapture { val harness = DeltaHarness.get - val (expected, expectedError, addFilesJson) = try { + val (expected, expectedError) = try { val log = harness.openLog(spark, tablePath.toString) val snapshot = JsonUtil.resolveSnapshot(spark, log, tablePath.toString, version, timestamp) @@ -65,13 +65,13 @@ object ReadCapture { try { writeExpectedData(expectedDir, df, count, specName) writeExpectedMetadata(spark, expectedDir, addFilesJson) - (Some(ReadExpected(count, addFilesJson.length, totalFileCount - addFilesJson.length)), None, addFilesJson) + (Some(ReadExpected(count, addFilesJson.length, totalFileCount - addFilesJson.length)), None) } finally { df.unpersist() } } catch { case e: Throwable => - (None, Some(SpecError(JsonUtil.extractErrorCode(e), Option(e.getMessage).getOrElse(""))), Seq.empty[String]) + (None, Some(SpecError(JsonUtil.extractErrorCode(e), Option(e.getMessage).getOrElse("")))) } // If the test author declared `expectError`, the operation MUST throw. @@ -99,7 +99,7 @@ object ReadCapture { } } - val spec = ReadSpec(version, timestamp, predicate, columns, expected, expectedError) + val spec = ReadSpec(writeSpec, version, timestamp, predicate, columns, expected, expectedError) JsonUtil.writeSpec(specPath, spec) validateFromSpec(spark, tablePath, expectedDir, specPath) @@ -132,7 +132,7 @@ object ReadCapture { } def validateFromSpec(spark: SparkSession, tablePath: Path, - expectedDir: Path, specPath: Path): Unit = { + expectedDir: Path, specPath: Path, checkMetadata: Boolean = true): Unit = { val spec = JsonUtil.readReadSpec(specPath) val specName = specPath.getFileName.toString.stripSuffix(".json") @@ -160,23 +160,30 @@ object ReadCapture { require(actualCode.isDefined, s"Error validation FAILED for $specName: expected operation to fail but it succeeded") } else if (spec.expected.isDefined) { + val exp = spec.expected.get val rereadDf = JsonUtil.applyFilters( JsonUtil.buildDeltaReader(spark, tablePath, spec.version, spec.timestamp), spec.predicate, spec.columns) - // A spec with `expected` MUST have captured expected_data. A missing directory means - // the corpus is incomplete (or one engine produced data and another did not), so fail - // loudly rather than silently skipping the row check. val expectedDataPath = expectedDir.resolve("expected_data") - require(Files.exists(expectedDataPath), - s"Validation FAILED for $specName: expected_data is missing") - JsonUtil.assertMultisetsEqual( - JsonUtil.toRowMultiset(spark.read.parquet(expectedDataPath.toString)), - JsonUtil.toRowMultiset(rereadDf), - specName) + if (Files.exists(expectedDataPath)) { + JsonUtil.assertMultisetsEqual( + JsonUtil.toRowMultiset(spark.read.parquet(expectedDataPath.toString)), + JsonUtil.toRowMultiset(rereadDf), + specName) + } else { + // expected_data is only skipped at capture when the row count exceeds the + // materialization limit (see writeExpectedData), so verify the row count for those + // specs. A missing dir at or under the limit means the corpus is incomplete: fail. + require(exp.rowCount > MaxExpectedDataRows, + s"Validation FAILED for $specName: expected_data is missing") + val actual = rereadDf.count() + require(actual == exp.rowCount, + s"Validation FAILED for $specName: row count $actual != expected ${exp.rowCount}") + } val expectedMetaPath = expectedDir.resolve("expected_metadata") - if (Files.exists(expectedMetaPath)) { + if (checkMetadata && Files.exists(expectedMetaPath)) { // Re-derive which AddFiles the current scan would touch, so we can // compare against what was captured at generation time. val harness = DeltaHarness.get diff --git a/workload-generator/src/main/scala/io/delta/workload/SnapshotCapture.scala b/workload-generator/src/main/scala/io/delta/workload/SnapshotCapture.scala index 9e07ccd5..a0fde145 100644 --- a/workload-generator/src/main/scala/io/delta/workload/SnapshotCapture.scala +++ b/workload-generator/src/main/scala/io/delta/workload/SnapshotCapture.scala @@ -27,7 +27,7 @@ object SnapshotCapture { def capture( spark: SparkSession, testId: String, tablePath: Path, specsDir: Path, version: Option[Long] = None, timestamp: Option[String] = None, - expectError: Option[String] = None): Unit = { + expectError: Option[String] = None, writeSpec: Option[String] = None): Unit = { require(!(version.isDefined && timestamp.isDefined), "Cannot specify both version and timestamp") @@ -79,7 +79,7 @@ object SnapshotCapture { } val specVersion = version.orElse(if (timestamp.isEmpty) resolvedVersion else None) - val spec = SnapshotSpec(specVersion, timestamp, expected, expectedError) + val spec = SnapshotSpec(writeSpec, specVersion, timestamp, expected, expectedError) JsonUtil.writeSpec(specPath, spec) validateFromSpec(spark, tablePath, specPath) diff --git a/workload-generator/src/main/scala/io/delta/workload/TableInfoWriter.scala b/workload-generator/src/main/scala/io/delta/workload/TableInfoWriter.scala index e8e59474..66a3ef39 100644 --- a/workload-generator/src/main/scala/io/delta/workload/TableInfoWriter.scala +++ b/workload-generator/src/main/scala/io/delta/workload/TableInfoWriter.scala @@ -24,6 +24,7 @@ import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions.{col, to_json} import io.delta.workload.deltaharness.DeltaHarness +import io.delta.workload.log.{CommitLog, RemoveFile} object TableInfoWriter { @@ -66,11 +67,12 @@ object TableInfoWriter { val numCommits = commitFiles.size - // Count actions and remove actions in a single pass over each commit. + // Count actions and remove actions in a single pass over each commit, via the typed log. val (numActions, numRemoveFiles) = commitFiles.foldLeft((0L, 0L)) { case ((actions, removes), f) => - val lines = Files.readAllLines(f).asScala - (actions + lines.size, removes + lines.count(_.startsWith("""{"remove"""))) + val version = f.getFileName.toString.stripSuffix(".json").toLong + val as = CommitLog.read(tablePath, version) + (actions + as.size, removes + as.count(_.isInstanceOf[RemoveFile])) } val lastCrcVersion = listLog(_.map(_.getFileName.toString) diff --git a/workload-generator/src/main/scala/io/delta/workload/WorkloadGenerator.scala b/workload-generator/src/main/scala/io/delta/workload/WorkloadGenerator.scala index 3a8f9b36..ece511e4 100644 --- a/workload-generator/src/main/scala/io/delta/workload/WorkloadGenerator.scala +++ b/workload-generator/src/main/scala/io/delta/workload/WorkloadGenerator.scala @@ -19,12 +19,13 @@ package io.delta.workload import java.nio.file.{Files, Path, Paths} import scala.collection.mutable -import scala.jdk.CollectionConverters._ -import com.fasterxml.jackson.databind.node.ObjectNode import org.apache.commons.io.FileUtils import org.apache.spark.sql.SparkSession -import io.delta.workload.deltaharness.DeltaHarness +import org.apache.spark.sql.types.StructType + +import io.delta.workload.deltaharness.{CommitRemoveFile, CommitRequest, DeltaHarness} +import io.delta.workload.log.{Action, CommitLog} /** * Internal workload generation engine. Use [[WorkloadTestSuite]] as the public API: @@ -72,15 +73,15 @@ object WorkloadGenerator { // Copy Delta table. Skip: // - transient files that async engine hooks may be mid-cleaning (e.g. - // some engines write `.crc..tmp` files and delete them shortly after — + // some engines write `.crc..tmp` files and delete them shortly after; // they can disappear between listing and copying) - // - Hadoop CRC sidecars (dot-prefixed `.NAME.crc`) — these would go + // - Hadoop CRC sidecars (dot-prefixed `.NAME.crc`): these would go // stale on any subsequent `mutateTable` / `modifyCommitActions` and // cause `ChecksumFileSystem` to reject the (intentionally) corrupted // file. Workloads use `RawLocalFileSystem` for reads anyway, so the // sidecars carry no information for our consumers. // NOTE: Delta version-checksums (`_delta_log/NNN.crc`, no dot prefix) are - // NOT excluded — those are protocol-level artifacts. + // NOT excluded; those are protocol-level artifacts. val destTablePath = testOutputDir.resolve("delta") val copyFilter: java.io.FileFilter = (f: java.io.File) => { val n = f.getName @@ -95,6 +96,10 @@ object WorkloadGenerator { mutate(destTablePath) } + // When this workload has a write spec, read/snapshot specs point to it (making them + // write-derived: a consumer reconstructs the table from the write spec before validating). + val writePtr = ts.writeBuilder.map(_ => s"${dirName}_write.json") + // Snapshot specs val snapshotNames = mutable.ArrayBuffer[String]() val explicits = if (ts.snapshotSpecs.isEmpty) { @@ -103,7 +108,7 @@ object WorkloadGenerator { for (ss <- explicits) { SnapshotCapture.capture(spark, dirName, destTablePath, specsDir, version = ss.version, timestamp = ss.timestamp, - expectError = ss.expectError) + expectError = ss.expectError, writeSpec = writePtr) val specName = (ss.version, ss.timestamp) match { case (Some(v), _) => s"${dirName}_snapshot_v$v" case (_, Some(t)) => @@ -120,12 +125,17 @@ object WorkloadGenerator { ReadCapture.capture(spark, dirName, destTablePath, testOutputDir, specsDir, name = rs.name, predicate = rs.predicate, version = rs.version, timestamp = rs.timestamp, columns = rs.columns, - expectError = rs.expectError) + expectError = rs.expectError, writeSpec = writePtr) val specName = s"${dirName}_${rs.name}" readNames += specName checkAssertion(rs, specsDir.resolve(s"$specName.json")) } + // Write spec - serialize the recorded commit history into specs/ + expected/_write/ + ts.writeBuilder.foreach { builder => + builder.buildSpec(spark, destTablePath, testOutputDir) + } + // table_info.json - wrap in try/catch for corrupt tables try { TableInfoWriter.write(spark, destTablePath, testOutputDir, @@ -135,6 +145,14 @@ object WorkloadGenerator { System.err.println(s"WARN: Could not write table_info.json for $dirName: ${e.getMessage}") } + // Self-validate the write workload: the single WorkloadValidator replays the write spec into + // a fresh table and re-checks the write-derived read/snapshot specs against it. + ts.writeBuilder.foreach { _ => + val result = WorkloadValidator.validateTestDir(spark, testOutputDir) + require(result.success, + s"Write spec validation FAILED for $dirName:\n ${result.errors.mkString("\n ")}") + } + // Repro placeholder val reproDir = testOutputDir.resolve("repro") Files.createDirectories(reproDir) @@ -150,10 +168,51 @@ object WorkloadGenerator { } +private[workload] trait HasAssertion[T] { + var assertion: Option[com.fasterxml.jackson.databind.JsonNode => Unit] = None + def deserialize: com.fasterxml.jackson.databind.JsonNode => T +} + // --------------------------------------------------------------------------- -// SpecRef — returned by spec declaration methods for optional assertions +// Internal data structures: the model the DSL builds and the engine consumes // --------------------------------------------------------------------------- +private[workload] class TableSpec( + private var _outputName: String, + val description: String, + val tags: Seq[String], + val sourcePath: Path) { + def outputName: String = _outputName + /** Set once by the orchestrator after body execution. */ + private[workload] def resolveOutputName(name: String): Unit = { _outputName = name } + val readSpecs = mutable.ArrayBuffer[ReadSpecConfig]() + val snapshotSpecs = mutable.ArrayBuffer[SnapshotSpecConfig]() + val mutations = mutable.ArrayBuffer[Path => Unit]() + var writeBuilder: Option[WriteSpecBuilder] = None +} + +case class WorkloadResult( + outputDir: String, + testId: String, + specsGenerated: Int, + readSpecs: Seq[String], + snapshotSpecs: Seq[String]) + +private[workload] case class ReadSpecConfig( + name: String, predicate: Option[String], version: Option[Long], + timestamp: Option[String], columns: Option[Seq[String]], + expectError: Option[String] = None) extends HasAssertion[ReadSpec] { + val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) => + JsonUtil.mapper.treeToValue(n, classOf[ReadSpec]) +} + +private[workload] case class SnapshotSpecConfig( + version: Option[Long], timestamp: Option[String], + expectError: Option[String] = None) extends HasAssertion[SnapshotSpec] { + val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) => + JsonUtil.mapper.treeToValue(n, classOf[SnapshotSpec]) +} + /** * Handle to a declared spec. Attach assertions that are checked after capture. * `T` is the typed spec class (e.g. [[ReadSpec]], [[SnapshotSpec]]). @@ -183,15 +242,6 @@ class SpecRef[T] private[workload] ( } } -private[workload] trait HasAssertion[T] { - var assertion: Option[com.fasterxml.jackson.databind.JsonNode => Unit] = None - def deserialize: com.fasterxml.jackson.databind.JsonNode => T -} - -// --------------------------------------------------------------------------- -// TableHandle — returned by registerTable() / registerTableFromPath() -// --------------------------------------------------------------------------- - /** * Handle to a Delta table created via SQL. Used for declaring read specs * (read, snapshot) and table mutations (mutateTable, modifyCommitActions). @@ -210,7 +260,7 @@ class TableHandle private[workload] ( * - Non-ICT tables: the commit JSON file's mtime. * * DESCRIBE HISTORY and `commitInfo.timestamp` both look right at a glance - * but diverge by a few milliseconds from the file mtime on some engines — feeding + * but diverge by a few milliseconds from the file mtime on some engines. Feeding * those back as `timestampAsOf` raises DELTA_TIMESTAMP_GREATER_THAN_COMMIT * / DELTA_TIMESTAMP_EARLIER_THAN_COMMIT_RETENTION. */ @@ -235,9 +285,16 @@ class TableHandle private[workload] ( } } -// --------------------------------------------------------------------------- -// WorkloadContext — the user's interface inside a workload() block -// --------------------------------------------------------------------------- +/** + * Handle for structured write operations. Obtained from [[WorkloadContext.createTableOp]] + * (which creates the table and records the create) or [[WorkloadContext.writeSpec]] + * (which wraps an already-created [[TableHandle]]). + * + * Accepts write operations (insertOp, updateOp, deleteOp, ...) that both execute SQL + * and record the commit. Call [[WorkloadContext.registerWriteSpec]] to finalize and + * obtain a [[TableHandle]] for declaring read/snapshot specs. + */ +class WriteHandle private[workload] (private[workload] val table: TableHandle) class WorkloadContext private[workload] ( val spark: SparkSession, @@ -250,6 +307,9 @@ class WorkloadContext private[workload] ( /** Convert nullable java.lang.Long to Option[Long]. */ private def opt(v: java.lang.Long): Option[Long] = Option(v).map(_.longValue()) + /** Wrap a possibly-empty collection: Some(it) when non-empty, else None. */ + private def opt[C <: Iterable[_]](c: C): Option[C] = if (c.nonEmpty) Some(c) else None + // Per-table state: spec names must be unique within each table private val _tableSpecNames = mutable.HashMap[String, mutable.HashSet[String]]() @@ -299,7 +359,7 @@ class WorkloadContext private[workload] ( * doesn't match. * - non-null + empty string `""`: any error is accepted; capture fails * only if the operation succeeds. - * - null (default): preserve legacy auto-detect behavior — record + * - null (default): preserve legacy auto-detect behavior, recording * whatever Spark does (success or error). */ def readSpec( @@ -346,6 +406,247 @@ class WorkloadContext private[workload] ( DeltaHarness.get.openLog(spark, loc).checkpoint() } + // ---- Write specs ---- + + /** ` PARTITIONED BY (a, b)` clause, or empty when there are no partition columns. */ + private def partitionedByClause(partitionColumns: Seq[String]): String = + if (partitionColumns.nonEmpty) s" PARTITIONED BY (${partitionColumns.mkString(", ")})" else "" + + /** ` TBLPROPERTIES ('k' = 'v', ...)` clause, or empty when there are no properties. */ + private def tblPropertiesClause(properties: Map[String, String]): String = + if (properties.nonEmpty) { + s" TBLPROPERTIES (${properties.map { case (k, v) => s"'$k' = '$v'" }.mkString(", ")})" + } else "" + + private val _writeBuilders = mutable.HashMap[String, WriteSpecBuilder]() + + private def getWriteBuilder(table: TableHandle): WriteSpecBuilder = { + val key = s"${workloadName}_${table.tableName}" + val builder = _writeBuilders.getOrElseUpdate(key, new WriteSpecBuilder()) + getTableSpec(table).writeBuilder = Some(builder) + builder + } + + /** Wrap an already-created [[TableHandle]] for structured write operations. */ + def writeSpec(table: TableHandle): WriteHandle = { + getWriteBuilder(table) + new WriteHandle(table) + } + + /** Finalize write operations and return the [[TableHandle]] for read/snapshot specs. */ + def registerWriteSpec(w: WriteHandle): TableHandle = w.table + + /** + * Create a table via SQL and record the create_table operation. `schema` is a SQL + * DDL string (e.g. "id INT, name STRING NOT NULL"). Returns a [[WriteHandle]] for + * further write operations. + */ + def createTableOp( + tableName: String, + schema: String, + properties: Map[String, String] = Map.empty, + partitionColumns: Seq[String] = Seq.empty): WriteHandle = { + val partitionClause = partitionedByClause(partitionColumns) + val propsClause = tblPropertiesClause(properties) + sql(s"CREATE TABLE $tableName ($schema) USING delta$partitionClause$propsClause") + val t = registerTable(tableName) + val b = getWriteBuilder(t) + b.record(CreateTableCommit( + schema = b.ddlToSchemaJson(schema), + partitionColumns = opt(partitionColumns), + properties = opt(properties))) + new WriteHandle(t) + } + + /** + * Replace an existing table's schema/partitioning/properties and all data via + * `CREATE OR REPLACE TABLE`, and record the replace_table operation. `schema` is a SQL DDL + * string. With non-empty `rows` it is a replace-as-select (RTAS): a single commit that also + * writes the rows. Operates on the table behind `w`; subsequent ops see the new schema. + */ + def replaceTableOp( + w: WriteHandle, + schema: String, + properties: Map[String, String] = Map.empty, + partitionColumns: Seq[String] = Seq.empty, + rows: Iterable[Map[String, Any]] = Seq.empty): Unit = { + val rowSeq = rows.iterator.toSeq + val partitionClause = partitionedByClause(partitionColumns) + val propsClause = tblPropertiesClause(properties) + if (rowSeq.nonEmpty) { + // Single-commit replace-as-select: write the rows to a temp Parquet and RTAS from it, so + // the resulting schema matches what the bundled spec Parquet will reproduce on replay. + val parquet = writeRowsToTemp(StructType.fromDDL(schema), rowSeq) + try { + sql(s"CREATE OR REPLACE TABLE ${w.table.tableName} USING delta$partitionClause" + + s"$propsClause AS SELECT * FROM parquet.`${parquet.toAbsolutePath}`") + } finally { + FileUtils.deleteDirectory(parquet.getParent.toFile) + } + } else { + sql(s"CREATE OR REPLACE TABLE ${w.table.tableName} ($schema) USING delta" + + s"$partitionClause$propsClause") + } + val b = getWriteBuilder(w.table) + b.record(ReplaceTableCommit( + schema = b.ddlToSchemaJson(schema), + partitionColumns = opt(partitionColumns), + properties = opt(properties)), Seq(rowSeq)) + } + + /** Write `rows` to a single Parquet file in a fresh temp dir; the caller deletes it after use. */ + private def writeRowsToTemp(schema: StructType, rows: Seq[Map[String, Any]]): Path = { + val dest = Files.createTempDirectory("row-parquet").resolve("part-00000.parquet") + DeltaHarness.get.writeRows(spark, schema, rows, dest) + dest + } + + /** + * Insert `rows` (column -> value maps) and record the insert. An empty `rows` is a no-op: + * it produces no commit, so recording it would both desync the commit-index/version mapping + * and let the validator pass a spec with nothing to validate. + */ + def insertOp(w: WriteHandle, rows: Iterable[Map[String, Any]]): Unit = { + val rowSeq = rows.iterator.toSeq + require(rowSeq.nonEmpty, "insertOp requires at least one row") + // Drive the live insert from the SAME materialized Parquet the spec will bundle (via the + // harness), so the captured table and the replayed table use one value encoder and + // cannot diverge by type. The spec's own copy is re-materialized in buildSpec. + val schema = DeltaHarness.get.schemaAt(spark, w.table.sourcePath.toString, version = None) + val parquet = writeRowsToTemp(schema, rowSeq) + try { + sql(s"INSERT INTO ${w.table.tableName} SELECT * FROM parquet.`${parquet.toAbsolutePath}`") + } finally { + FileUtils.deleteDirectory(parquet.getParent.toFile) + } + getWriteBuilder(w.table).record(InsertCommit(), Seq(rowSeq)) + } + + /** Delete rows matching `predicate` and record the delete. */ + def deleteOp(w: WriteHandle, predicate: String): Unit = { + sql(s"DELETE FROM ${w.table.tableName} WHERE $predicate") + getWriteBuilder(w.table).record(DeleteCommit(predicate)) + } + + /** Update rows matching `predicate` with `set` (column -> expression) and record it. */ + def updateOp(w: WriteHandle, predicate: String, set: Map[String, String]): Unit = { + val setClauses = set.map { case (k, v) => s"`$k` = $v" }.mkString(", ") + sql(s"UPDATE ${w.table.tableName} SET $setClauses WHERE $predicate") + getWriteBuilder(w.table).record(UpdateCommit(predicate, set)) + } + + + /** + * Execute a low-level commit of raw Delta actions against the live table and record it. `addFiles` + * supply LOGICAL rows; the engine writes them (column-mapping/partition/stats handled) and the + * actions are bundled with the raw `txn`/`addDomainMetadata`/`removeDomainMetadata`/schema/property + * changes in one commit. `removeFiles` reference a prior low-level add by the ordinal that + * [[commitOp]] returned (the commit/version that produced it). Returns this commit's ordinal so a + * later `removeFiles` can target it. + */ + def commitOp( + w: WriteHandle, + schemaDDL: Option[String] = None, + tableProperties: Option[Map[String, String]] = None, + txn: Option[AppTxn] = None, + addFiles: Option[Seq[AddFileInput]] = None, + removeFiles: Option[Seq[Int]] = None, + addDomainMetadata: Option[Seq[AddDomainMetadata]] = None, + removeDomainMetadata: Option[Seq[String]] = None): Int = { + require( + schemaDDL.isDefined || tableProperties.exists(_.nonEmpty) || txn.isDefined || + addFiles.exists(_.nonEmpty) || removeFiles.exists(_.nonEmpty) || + addDomainMetadata.exists(_.nonEmpty) || removeDomainMetadata.exists(_.nonEmpty), + "commitOp requires at least one action (schema/properties/txn/addFiles/removeFiles/" + + "domainMetadata); an empty commit would not advance the table version") + val builder = getWriteBuilder(w.table) + val idx = builder.nextOrdinal + val livePath = w.table.sourcePath.toString + val adds = addFiles.getOrElse(Seq.empty) + + // Materialize each add's logical rows to a temp Parquet for the live write. + val schema = DeltaHarness.get.schemaAt(spark, livePath, version = None) + val tempParquet = adds.map(in => writeRowsToTemp(schema, in.rows)) + // A remove targets all files added at the referenced commit (== that table version). + val removePaths = removeFiles.getOrElse(Seq.empty) + .flatMap(k => SpecLayout.addPathsAt(w.table.sourcePath, k)) + try { + DeltaHarness.get.commit(spark, livePath, + CommitRequest( + addDataParquet = tempParquet.map(_.toAbsolutePath.toString), + schemaJson = schemaDDL.map(ddl => StructType.fromDDL(ddl).json), + properties = tableProperties, + setTransaction = txn, + removeFiles = removePaths.map(p => CommitRemoveFile(p, dataChange = true)), + addDomainMetadata = addDomainMetadata.getOrElse(Seq.empty), + removeDomainMetadata = removeDomainMetadata.getOrElse(Seq.empty))) + } finally { + tempParquet.foreach(p => FileUtils.deleteDirectory(p.getParent.toFile)) + } + + // Enforce the load-bearing commit-index == table-version invariant at the point it is + // established: ordinal-based remove resolution (SpecLayout.addPathsAt) keys off it. Fail here, + // at the offending op, rather than deferring to buildSpec's whole-spec count check. + val newVersion = DeltaHarness.get.openLog(spark, livePath).update().version + require(newVersion == idx, + s"commitOp produced version $newVersion but expected ordinal $idx; commit index can no " + + "longer be used as the table version (ordinal-based removes would resolve incorrectly)") + + // Record the spec: adds become logical-Parquet pointers (materialized to data/ in buildSpec); + // removes become ordinal references. + val addActions = adds.zipWithIndex.map { case (in, i) => + AddFileAction(dataFile = SpecLayout.commitDataFile(idx, s"add_$i.parquet"), dataChange = in.dataChange) + } + builder.record(LowLevelCommitOp( + schema = schemaDDL.map(builder.ddlToSchemaJson), + tableProperties = tableProperties, txn = txn, + addFiles = opt(addActions), + removeFiles = removeFiles.map(_.map(k => RemoveFileAction(k))), + addDomainMetadata = addDomainMetadata, removeDomainMetadata = removeDomainMetadata), + adds.map(_.rows)) + idx + } + + /** Add columns (SQL DDL) and record the schema evolution. */ + def addColumnsOp(w: WriteHandle, columnsDDL: String): Unit = { + require(columnsDDL.nonEmpty, "addColumnsOp requires a non-empty DDL string") + sql(s"ALTER TABLE ${w.table.tableName} ADD COLUMNS ($columnsDDL)") + val b = getWriteBuilder(w.table) + b.record(EvolveSchemaCommit(addColumns = b.addColumnsJson(columnsDDL))) + } + + /** Rename a column and record the schema evolution. */ + def renameColumnOp(w: WriteHandle, oldName: String, newName: String): Unit = { + sql(s"ALTER TABLE ${w.table.tableName} RENAME COLUMN $oldName TO $newName") + getWriteBuilder(w.table).record(EvolveSchemaCommit(renameColumns = Some(Map(oldName -> newName)))) + } + + /** Drop columns and record the schema evolution. */ + def dropColumnsOp(w: WriteHandle, columns: Seq[String]): Unit = { + require(columns.nonEmpty, "dropColumnsOp requires at least one column") + if (columns.size == 1) { + sql(s"ALTER TABLE ${w.table.tableName} DROP COLUMN ${columns.head}") + } else { + sql(s"ALTER TABLE ${w.table.tableName} DROP COLUMNS (${columns.mkString(", ")})") + } + getWriteBuilder(w.table).record(EvolveSchemaCommit(dropColumns = opt(columns))) + } + + /** Set table properties and record the update_properties operation. */ + def setPropertiesOp(w: WriteHandle, props: Map[String, String]): Unit = { + require(props.nonEmpty, "setPropertiesOp requires at least one property") + val setClause = props.map { case (k, v) => s"'$k' = '$v'" }.mkString(", ") + sql(s"ALTER TABLE ${w.table.tableName} SET TBLPROPERTIES ($setClause)") + getWriteBuilder(w.table).record(UpdatePropertiesCommit(set = Some(props))) + } + + /** Unset table properties and record the update_properties operation. */ + def unsetPropertiesOp(w: WriteHandle, props: Seq[String]): Unit = { + require(props.nonEmpty, "unsetPropertiesOp requires at least one property") + sql(s"ALTER TABLE ${w.table.tableName} UNSET TBLPROPERTIES (${props.map(k => s"'$k'").mkString(", ")})") + getWriteBuilder(w.table).record(UpdatePropertiesCommit(remove = Some(props))) + } + // ---- Table mutations (applied to copied table before spec capture) ---- /** Mutate the copied table's filesystem before specs are captured. */ @@ -354,33 +655,16 @@ class WorkloadContext private[workload] ( } /** - * Modify actions in a specific commit version. - * The modifier receives the full list of `(actionType, innerNode)` pairs - * (e.g. `("add", )`) and returns the (possibly reordered, - * filtered, or modified) list to write back. + * Modify the typed [[Action]]s of a specific commit version: filter, reorder, or `.copy` fields. + * Unknown/malformed lines surface as [[io.delta.workload.log.RawAction]] and pass through + * unchanged. For byte-level corruption (truncation, garbage), use [[mutateTable]] directly. */ def modifyCommitActions(table: TableHandle, version: Long)( - modifier: Seq[(String, ObjectNode)] => Seq[(String, ObjectNode)]): Unit = { + modifier: Seq[Action] => Seq[Action]): Unit = mutateTable(table) { tableDir => - val commitFile = tableDir.resolve("_delta_log").resolve(f"$version%020d.json") - if (Files.exists(commitFile)) { - val lines = new String(Files.readAllBytes(commitFile), "UTF-8").split("\n") - .filter(_.trim.nonEmpty) - val actions = lines.map { line => - val node = JsonUtil.mapper.readTree(line) - val actionType = node.fieldNames().next() - (actionType, node.get(actionType).asInstanceOf[ObjectNode]) - }.toSeq - val result = modifier(actions) - val newLines = result.map { case (actionType, innerNode) => - val wrapper = JsonUtil.mapper.createObjectNode() - wrapper.set(actionType, innerNode) - JsonUtil.mapper.writeValueAsString(wrapper) - } - Files.write(commitFile, newLines.mkString("\n").getBytes("UTF-8")) - } + if (Files.exists(CommitLog.commitFile(tableDir, version))) + CommitLog.mutate(tableDir, version)(modifier) } - } // ---- Auto-naming ---- @@ -515,10 +799,6 @@ object WorkloadContext { def withContext[T](ctx: WorkloadContext)(body: => T): T = _current.withValue(ctx)(body) } -// --------------------------------------------------------------------------- -// WorkloadOps — DSL trait mixed into WorkloadSuite for clean syntax -// --------------------------------------------------------------------------- - /** * Provides DSL methods for workload generation. Mixed into WorkloadSuite * so that test bodies can call methods like `sql()`, `registerTable()`, `readSpec()` @@ -567,51 +847,82 @@ trait WorkloadOps { /** Force Spark to write a checkpoint file for the given SQL table name. */ def forceCheckpoint(tableName: String): Unit = current.forceCheckpoint(tableName) + // ---- Write specs ---- + + /** Wrap an already-created [[TableHandle]] for structured write operations. */ + def writeSpec(table: TableHandle): WriteHandle = current.writeSpec(table) + + /** Finalize write operations and return the [[TableHandle]] for read/snapshot specs. */ + def registerWriteSpec(w: WriteHandle): TableHandle = current.registerWriteSpec(w) + + /** Create a table via SQL and record the create_table operation. */ + def createTableOp( + tableName: String, + schema: String, + properties: Map[String, String] = Map.empty, + partitionColumns: Seq[String] = Seq.empty): WriteHandle = + current.createTableOp(tableName, schema, properties, partitionColumns) + + /** Replace an existing table (schema/partitioning/properties + data) and record it. */ + def replaceTableOp( + w: WriteHandle, + schema: String, + properties: Map[String, String] = Map.empty, + partitionColumns: Seq[String] = Seq.empty, + rows: Iterable[Map[String, Any]] = Seq.empty): Unit = + current.replaceTableOp(w, schema, properties, partitionColumns, rows) + + /** Insert rows and record the insert. */ + def insertOp(w: WriteHandle, rows: Iterable[Map[String, Any]]): Unit = + current.insertOp(w, rows) + + /** Delete rows matching `predicate` and record the delete. */ + def deleteOp(w: WriteHandle, predicate: String): Unit = current.deleteOp(w, predicate) + + /** Update rows matching `predicate` with `set` and record the update. */ + def updateOp(w: WriteHandle, predicate: String, set: Map[String, String]): Unit = + current.updateOp(w, predicate, set) + + + /** Execute a low-level commit of raw Delta actions and record it. Returns the commit ordinal. */ + def commitOp( + w: WriteHandle, + schemaDDL: Option[String] = None, + tableProperties: Option[Map[String, String]] = None, + txn: Option[AppTxn] = None, + addFiles: Option[Seq[AddFileInput]] = None, + removeFiles: Option[Seq[Int]] = None, + addDomainMetadata: Option[Seq[AddDomainMetadata]] = None, + removeDomainMetadata: Option[Seq[String]] = None): Int = + current.commitOp(w, schemaDDL, tableProperties, txn, addFiles, removeFiles, + addDomainMetadata, removeDomainMetadata) + + /** Add columns (SQL DDL) and record the schema evolution. */ + def addColumnsOp(w: WriteHandle, columnsDDL: String): Unit = + current.addColumnsOp(w, columnsDDL) + + /** Rename a column and record the schema evolution. */ + def renameColumnOp(w: WriteHandle, oldName: String, newName: String): Unit = + current.renameColumnOp(w, oldName, newName) + + /** Drop columns and record the schema evolution. */ + def dropColumnsOp(w: WriteHandle, columns: Seq[String]): Unit = + current.dropColumnsOp(w, columns) + + /** Set table properties and record the update_properties operation. */ + def setPropertiesOp(w: WriteHandle, props: Map[String, String]): Unit = + current.setPropertiesOp(w, props) + + /** Unset table properties and record the update_properties operation. */ + def unsetPropertiesOp(w: WriteHandle, props: Seq[String]): Unit = + current.unsetPropertiesOp(w, props) + /** Mutate the copied table's filesystem before specs are captured. */ def mutateTable(table: TableHandle)(mutation: Path => Unit): Unit = current.mutateTable(table)(mutation) - /** Modify actions in a specific commit version. */ + /** Modify the typed actions of a specific commit version. */ def modifyCommitActions(table: TableHandle, version: Long)( - modifier: Seq[(String, ObjectNode)] => Seq[(String, ObjectNode)]): Unit = + modifier: Seq[Action] => Seq[Action]): Unit = current.modifyCommitActions(table, version)(modifier) } - -// --------------------------------------------------------------------------- -// Internal data structures -// --------------------------------------------------------------------------- - -private[workload] class TableSpec( - private var _outputName: String, - val description: String, - val tags: Seq[String], - val sourcePath: Path) { - def outputName: String = _outputName - /** Set once by the orchestrator after body execution. */ - private[workload] def resolveOutputName(name: String): Unit = { _outputName = name } - val readSpecs = mutable.ArrayBuffer[ReadSpecConfig]() - val snapshotSpecs = mutable.ArrayBuffer[SnapshotSpecConfig]() - val mutations = mutable.ArrayBuffer[Path => Unit]() -} - -case class WorkloadResult( - outputDir: String, - testId: String, - specsGenerated: Int, - readSpecs: Seq[String], - snapshotSpecs: Seq[String]) - -private[workload] case class ReadSpecConfig( - name: String, predicate: Option[String], version: Option[Long], - timestamp: Option[String], columns: Option[Seq[String]], - expectError: Option[String] = None) extends HasAssertion[ReadSpec] { - val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) => - JsonUtil.mapper.treeToValue(n, classOf[ReadSpec]) -} - -private[workload] case class SnapshotSpecConfig( - version: Option[Long], timestamp: Option[String], - expectError: Option[String] = None) extends HasAssertion[SnapshotSpec] { - val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) => - JsonUtil.mapper.treeToValue(n, classOf[SnapshotSpec]) -} diff --git a/workload-generator/src/main/scala/io/delta/workload/WorkloadTestSuite.scala b/workload-generator/src/main/scala/io/delta/workload/WorkloadTestSuite.scala index 1960960b..5337aa53 100644 --- a/workload-generator/src/main/scala/io/delta/workload/WorkloadTestSuite.scala +++ b/workload-generator/src/main/scala/io/delta/workload/WorkloadTestSuite.scala @@ -18,7 +18,6 @@ package io.delta.workload import java.nio.file.{Files, Path, Paths} -import org.apache.commons.io.FileUtils import org.apache.spark.sql.SparkSession import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach} import org.scalatest.funsuite.AnyFunSuite @@ -178,8 +177,4 @@ abstract class WorkloadTestSuite(override val suiteName: String) } } } - - private def cleanupDir(dir: Path): Unit = { - if (Files.exists(dir)) FileUtils.deleteDirectory(dir.toFile) - } } diff --git a/workload-generator/src/main/scala/io/delta/workload/WorkloadValidator.scala b/workload-generator/src/main/scala/io/delta/workload/WorkloadValidator.scala index 8755f084..664e77ea 100644 --- a/workload-generator/src/main/scala/io/delta/workload/WorkloadValidator.scala +++ b/workload-generator/src/main/scala/io/delta/workload/WorkloadValidator.scala @@ -17,11 +17,21 @@ package io.delta.workload import java.nio.file.{Files, Path} +import java.util.concurrent.Executors -import scala.collection.JavaConverters._ import scala.collection.mutable +import scala.concurrent.{Await, ExecutionContext, Future} +import scala.concurrent.duration.Duration +import scala.jdk.CollectionConverters._ +import scala.util.control.NonFatal -import org.apache.spark.sql.SparkSession +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.node.{ArrayNode, ObjectNode} +import org.apache.commons.io.FileUtils +import org.apache.spark.sql.{SaveMode, SparkSession} +import org.apache.spark.sql.types.{DataType, StructField, StructType} + +import io.delta.workload.deltaharness.{CommitRemoveFile, CommitRequest, DeltaHarness} /** Result of validating a workload tree: per-spec pass/fail tallies + error details. */ case class ValidationResult(passed: Int, errors: Seq[String]) { @@ -29,73 +39,396 @@ case class ValidationResult(passed: Int, errors: Seq[String]) { } /** - * Walks a generated workload tree and re-validates every captured spec against - * the tables on disk. Used as the presubmit acceptance-test entry point: given - * a committed tarball extracted locally, verify that the current Delta - * implementation can still reproduce the captured specifications. + * The single validator for every spec type. Walks a generated workload tree (or one test dir) + * and validates each captured spec under `specs/`, dispatching on the `type` field: * - * Expected tree layout (produced by [[WorkloadGenerator]]): - * {{{ - * // - * delta/ # the Delta table - * specs/.json # flat — dispatched by the "type" field - * expected// - * expected_data # parquet - * expected_metadata # parquet - * }}} + * - `write`: replay its `commits` into a fresh table, compare rows to `expected//`. + * - `read` / `snapshot`: if the spec carries a `writeSpec` pointer it is *write-derived*: + * reconstruct the table by replaying that sibling write spec, then validate a read spec by + * rows only; a write-derived snapshot only confirms the write spec replays (its + * protocol/metadata were validated against the real table at capture time). Otherwise it is + * *read-only*: validate against the captured `delta/` table exactly. * - * Each `/delta/` is opened and each spec under `/specs/` - * is replayed via [[ReadCapture.validateFromSpec]] or - * [[SnapshotCapture.validateFromSpec]] based on the spec's `type` field. - * A missing `delta/` is skipped (corruption tests may omit the table). + * Used both as the presubmit acceptance entry point ([[validateAll]]) and by the generator to + * self-check a freshly generated test dir ([[validateTestDir]]). */ object WorkloadValidator { def validateAll(spark: SparkSession, workloadRoot: Path): ValidationResult = { require(Files.isDirectory(workloadRoot), s"Not a directory: $workloadRoot") + val testDirs = listChildren(workloadRoot).filter(Files.isDirectory(_)) + // Each test dir validates independently (its own temp replay tables, path-based table refs), + // and SparkSession is thread-safe, so validate them concurrently and let Spark's scheduler + // interleave the per-dir jobs. (DeltaLog's global cache only thrashes under this; commits are + // per-table, so the result stays correct.) + val pool = Executors.newFixedThreadPool( + math.min(8, math.max(1, Runtime.getRuntime.availableProcessors))) + implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(pool) + val results = + try Await.result(Future.sequence(testDirs.map(d => Future(validateTestDir(spark, d)))), Duration.Inf) + finally pool.shutdown() + ValidationResult(results.map(_.passed).sum, results.flatMap(_.errors)) + } + + /** + * Validate every spec in one test dir. Write-derived read/snapshot specs and the write spec + * share a single replayed table per write spec (memoized). Returns pass count + error details. + */ + def validateTestDir(spark: SparkSession, testDir: Path): ValidationResult = { + val specsDir = testDir.resolve("specs") + if (!Files.isDirectory(specsDir)) return ValidationResult(0, Seq.empty) + val deltaTable = testDir.resolve("delta") val errors = mutable.ArrayBuffer[String]() var passed = 0 + val replays = mutable.Map[String, Path]() // write-spec file name -> replayed table path + val tempDirs = mutable.ArrayBuffer[Path]() + def replayOf(writeSpecName: String): Path = replays.getOrElseUpdate(writeSpecName, { + val td = Files.createTempDirectory("wl_replay") + tempDirs += td + val table = td.resolve("replay_table") + replayWriteInto(spark, testDir, specsDir.resolve(writeSpecName), table) + table + }) - val testDirs = listChildren(workloadRoot).filter(Files.isDirectory(_)) + try { + for (specFile <- listSpecs(specsDir)) { + val name = specFile.getFileName.toString.stripSuffix(".json") + val node = JsonUtil.mapper.readTree(Files.readAllBytes(specFile)) + val specType = Option(node.get("type")).map(_.asText()).getOrElse("") + val writePtr = Option(node.get("writeSpec")).filterNot(_.isNull).map(_.asText()) - for (testDir <- testDirs) { - val tablePath = testDir.resolve("delta") - val specsDir = testDir.resolve("specs") - if (!Files.isDirectory(tablePath) || !Files.isDirectory(specsDir)) { - // Nothing to validate — corruption tests may legitimately omit these. - } else { - for (specPath <- listChildren(specsDir).filter(_.getFileName.toString.endsWith(".json"))) { - val specName = specPath.getFileName.toString.stripSuffix(".json") - val expectedDir = testDir.resolve("expected").resolve(specName) - try { - dispatch(spark, tablePath, expectedDir, specPath) + // Read-only specs validate against the captured `delta/` table; if it is absent the corpus + // is incomplete. Run `validate` against it (counting a pass), else record the missing table. + def validateReadOnly(validate: => Unit): Unit = + if (Files.isDirectory(deltaTable)) { + validate passed += 1 - } catch { - case e: Throwable => - errors += s"${testDir.getFileName}/$specName: ${e.getMessage}" + } else { + errors += s"${testDir.getFileName}/$name: read-only spec but no captured " + + s"table at $deltaTable" + } + + try { + specType match { + case "write" => + validateExpectedLatest(spark, replayOf(specFile.getFileName.toString), testDir, name) + passed += 1 + + case "read" => + val expectedDir = testDir.resolve("expected").resolve(name) + writePtr match { + case Some(ws) => // write-derived: rows-only against the replayed table + ReadCapture.validateFromSpec(spark, replayOf(ws), expectedDir, specFile, + checkMetadata = false) + passed += 1 + case None => // read-only: exact, against the captured table + validateReadOnly(ReadCapture.validateFromSpec(spark, deltaTable, expectedDir, specFile)) + } + + case "snapshot" => + writePtr match { + case Some(ws) => // write-derived: validate the replayed table's snapshot + validateSnapshotDerived(spark, replayOf(ws), specFile, + JsonUtil.readWriteSpec(specsDir.resolve(ws))) + passed += 1 + case None => + validateReadOnly(SnapshotCapture.validateFromSpec(spark, deltaTable, specFile)) + } + + case other => throw new IllegalArgumentException( + s"Unknown spec type '$other' in ${specFile.getFileName}") } + } catch { + case e: Throwable => errors += s"${testDir.getFileName}/$name: $e" } } + } finally { + tempDirs.foreach(td => try FileUtils.deleteDirectory(td.toFile) catch { case NonFatal(_) => }) } - ValidationResult(passed, errors.toSeq) } - /** Dispatch to the correct validator based on the spec's `type` field. */ - private def dispatch( - spark: SparkSession, tablePath: Path, expectedDir: Path, specPath: Path): Unit = { - val node = JsonUtil.mapper.readTree(Files.readAllBytes(specPath)) - Option(node.get("type")).map(_.asText()).getOrElse("") match { - case "read" => ReadCapture.validateFromSpec(spark, tablePath, expectedDir, specPath) - case "snapshot" => SnapshotCapture.validateFromSpec(spark, tablePath, specPath) - case other => throw new IllegalArgumentException( - s"Unknown spec type '$other' in ${specPath.getFileName}") - } + private def listSpecs(specsDir: Path): Seq[Path] = { + val stream = Files.list(specsDir) + try stream.iterator().asScala.filter(_.getFileName.toString.endsWith(".json")).toSeq.sorted + finally stream.close() } private def listChildren(dir: Path): List[Path] = { val stream = Files.list(dir) try stream.iterator().asScala.toList finally stream.close() } + + // =========================================================================== + // Write-spec replay (reconstruct a fresh table from a write spec's commits) + // =========================================================================== + + /** Replay a write spec's commits into a fresh table at `replayTablePath`. */ + private def replayWriteInto( + spark: SparkSession, testDir: Path, writeSpecFile: Path, replayTablePath: Path): Unit = { + val writeSpec = JsonUtil.readWriteSpec(writeSpecFile) + val ref = s"delta.`${replayTablePath.toAbsolutePath}`" + val path = replayTablePath.toAbsolutePath.toString + writeSpec.commits.zipWithIndex.foreach { case (c, idx) => + replayCommit(spark, c, idx, ref, path, testDir) + } + // Each commit must map to exactly one version, or ordinal-based remove resolution (which keys + // off commit index == table version) silently resolves the wrong version. Fail loud if desynced. + val finalVersion = DeltaHarness.get.openLog(spark, path).update().version + require(finalVersion == writeSpec.commits.size - 1, + s"replay produced ${finalVersion + 1} versions but the write spec has ${writeSpec.commits.size} " + + "commits; commit index can no longer be used as the table version") + } + + /** Compare the replayed table's rows against the write spec's `expected//table_content`. */ + private def validateExpectedLatest( + spark: SparkSession, replayTablePath: Path, testDir: Path, writeName: String): Unit = { + val contentDir = testDir.resolve("expected").resolve(writeName).resolve("table_content") + // A write spec MUST ship its expected rows; a missing dir is a capture bug. + require(Files.exists(contentDir), + s"write spec '$writeName': expected/$writeName/table_content is missing") + val expected = JsonUtil.toRowMultiset(spark.read.parquet(contentDir.toString)) + val actual = JsonUtil.toRowMultiset( + spark.read.format("delta").load(replayTablePath.toAbsolutePath.toString)) + JsonUtil.assertMultisetsEqual(expected, actual, s"${writeName}_latest") + } + + /** + * Validate a write-derived snapshot spec against the table replayed from its write spec: the + * replay must reproduce the captured `protocol` and `metadata`. Protocol is compared exactly + * (replay is Spark from the same commits, so it is deterministic). For metadata we compare the + * fields that are reproducible across an independently-created table: `schemaString` (with + * per-field column-mapping ids stripped, since those are minted per table), `partitionColumns`, + * `format`, and the author-declared `configuration` keys (engine-injected defaults, table `id`, + * and `createdTime` differ between table instances and are not compared). + */ + private def validateSnapshotDerived( + spark: SparkSession, replayTablePath: Path, specFile: Path, writeSpec: WriteSpec): Unit = { + val spec = JsonUtil.readSnapshotSpec(specFile) + val name = specFile.getFileName.toString.stripSuffix(".json") + val (declaredKeys, removedKeys) = declaredConfiguration(writeSpec) + spec.expected.foreach { exp => + val log = DeltaHarness.get.openLog(spark, replayTablePath.toString) + val snapshot = (spec.version, spec.timestamp) match { + case (Some(v), _) => log.getSnapshotAt(v) + case _ => log.update() + } + val expProtocol = JsonUtil.mapper.valueToTree[JsonNode](exp.protocol) + val replayProtocol = JsonUtil.mapper.readTree(snapshot.protocolJson).get("protocol") + require(JsonUtil.canonicalJson(expProtocol) == JsonUtil.canonicalJson(replayProtocol), + s"snapshot '$name': protocol mismatch (expected ${JsonUtil.canonicalJson(expProtocol)}, " + + s"got ${JsonUtil.canonicalJson(replayProtocol)})") + val expectedMeta = JsonUtil.mapper.valueToTree[JsonNode](exp.metadata) + val replayMeta = JsonUtil.mapper.readTree(snapshot.metadataJson).get("metaData") + configurationViolation(expectedMeta, replayMeta, declaredKeys, removedKeys).foreach { reason => + throw new IllegalStateException(s"snapshot '$name': metadata.configuration mismatch ($reason)") + } + require(normalizedSchema(expectedMeta) == normalizedSchema(replayMeta), + s"snapshot '$name': metadata.schemaString mismatch") + for (field <- Seq("partitionColumns", "format")) + require(canonicalField(expectedMeta, field) == canonicalField(replayMeta, field), + s"snapshot '$name': metadata.$field mismatch") + } + } + + /** The config keys the write spec's commits declared: (still-in-effect, removed). */ + private def declaredConfiguration(writeSpec: WriteSpec): (Set[String], Set[String]) = { + val declared = mutable.LinkedHashSet[String]() + val removed = mutable.LinkedHashSet[String]() + for (commit <- writeSpec.commits) commit match { + case c: CreateTableCommit => c.properties.foreach(declared ++= _.keys) + case c: ReplaceTableCommit => + declared.clear(); removed.clear() + c.properties.foreach(declared ++= _.keys) + case c: UpdatePropertiesCommit => + c.set.foreach { s => declared ++= s.keys; removed --= s.keys } + c.remove.foreach { r => removed ++= r; declared --= r } + case _ => () + } + (declared.toSet, removed.toSet) + } + + /** Each declared key present + equal on the replay; each removed key absent. First violation. */ + private def configurationViolation( + expectedMeta: JsonNode, replayMeta: JsonNode, + declaredKeys: Set[String], removedKeys: Set[String]): Option[String] = { + val expConfig = Option(expectedMeta.get("configuration")).filterNot(_.isNull) + val repConfig = Option(replayMeta.get("configuration")).filterNot(_.isNull) + def get(config: Option[JsonNode], key: String): Option[String] = + config.flatMap(c => Option(c.get(key))).filterNot(_.isNull).map(_.asText()) + declaredKeys.toSeq.sorted.collectFirst { + case key if get(repConfig, key) != get(expConfig, key) => + s"key '$key' expected ${get(expConfig, key)}, got ${get(repConfig, key)}" + }.orElse { + removedKeys.toSeq.sorted.collectFirst { + case key if get(repConfig, key).isDefined => s"removed key '$key' still present" + } + } + } + + /** + * Canonicalize `metadata.schemaString`, stripping per-field column-mapping identifiers + * (`delta.columnMapping.physicalName`/`.id`) which are minted per table and so are not a + * meaningful equality target across an independently-replayed table. + */ + private def normalizedSchema(meta: JsonNode): String = + Option(meta.get("schemaString")).filterNot(_.isNull) match { + case None => "null" + case Some(s) => + val tree = JsonUtil.mapper.readTree(s.asText()) + stripColumnMapping(tree) + JsonUtil.canonicalJson(tree) + } + + private def stripColumnMapping(node: JsonNode): Unit = node match { + case obj: ObjectNode => + obj.get("metadata") match { + case m: ObjectNode => + m.remove("delta.columnMapping.physicalName") + m.remove("delta.columnMapping.id") + case _ => () + } + obj.elements().asScala.foreach(stripColumnMapping) + case arr: ArrayNode => arr.elements().asScala.foreach(stripColumnMapping) + case _ => () + } + + private def canonicalField(node: JsonNode, field: String): String = + if (node.has(field) && !node.get(field).isNull) JsonUtil.canonicalJson(node.get(field)) + else "null" + + // =========================================================================== + // Per-commit replay + // =========================================================================== + + private def replayCommit( + spark: SparkSession, commit: WriteCommit, idx: Int, + tableRef: String, tablePath: String, testDir: Path): Unit = commit match { + case c: CreateTableCommit => + var sql = s"CREATE TABLE $tableRef (${structOf(c.schema).toDDL}) USING delta" + c.partitionColumns.filter(_.nonEmpty).foreach { p => + sql += s" PARTITIONED BY (${p.mkString(", ")})" + } + c.properties.filter(_.nonEmpty).foreach { props => + sql += s" TBLPROPERTIES (${props.map { case (k, v) => s"'$k' = '$v'" }.mkString(", ")})" + } + spark.sql(sql) + + case c: ReplaceTableCommit => + val parts = c.partitionColumns.filter(_.nonEmpty) + .map(p => s" PARTITIONED BY (${p.mkString(", ")})").getOrElse("") + val props = c.properties.filter(_.nonEmpty) + .map(m => s" TBLPROPERTIES (${m.map { case (k, v) => s"'$k' = '$v'" }.mkString(", ")})") + .getOrElse("") + // Always replace via AS SELECT: `CREATE OR REPLACE TABLE delta.`path` (cols)` is treated as + // create-and-validate by Delta; the query path honors the new schema. Data-less = 0 rows. + val select = c.dataFiles.filter(_.nonEmpty) match { + case Some(files) => + val resolved = files.map(f => testDir.resolve(f).toAbsolutePath.toString) + val src = if (resolved.size == 1) s"parquet.`${resolved.head}`" + else s"parquet.`${testDir.resolve(SpecLayout.commitDataDir(idx)).toAbsolutePath}`" + s"SELECT * FROM $src" + case None => + val nulls = structOf(c.schema).fields + .map(f => s"CAST(NULL AS ${f.dataType.sql}) AS `${f.name}`").mkString(", ") + s"SELECT * FROM (SELECT $nulls) WHERE false" + } + spark.sql(s"CREATE OR REPLACE TABLE $tableRef USING delta$parts$props AS $select") + + case c: InsertCommit => + c.dataFiles.foreach(loadDataFiles(spark, _, tablePath, testDir)) + + case c: DeleteCommit => + spark.sql(s"DELETE FROM $tableRef WHERE ${c.predicate}") + + case c: UpdateCommit => + val setClauses = c.set.map { case (k, v) => s"`$k` = $v" }.mkString(", ") + spark.sql(s"UPDATE $tableRef SET $setClauses WHERE ${c.predicate}") + + case c: UpdatePropertiesCommit => + // SET and UNSET replay as separate commits, so one UpdatePropertiesCommit must not do both + // (it would advance the table by two versions and break commit index == version). + require(!(c.set.exists(_.nonEmpty) && c.remove.exists(_.nonEmpty)), + s"UpdatePropertiesCommit at commit $idx both sets and removes properties") + c.set.filter(_.nonEmpty).foreach { s => + val setClause = s.map { case (k, v) => s"'$k' = '$v'" }.mkString(", ") + spark.sql(s"ALTER TABLE $tableRef SET TBLPROPERTIES ($setClause)") + } + c.remove.filter(_.nonEmpty).foreach { r => + spark.sql(s"ALTER TABLE $tableRef UNSET TBLPROPERTIES (${r.map(k => s"'$k'").mkString(", ")})") + } + + case c: EvolveSchemaCommit => replayEvolveSchema(spark, c, tableRef) + + case c: LowLevelCommitOp => replayLowLevelCommit(spark, c, idx, tablePath, testDir) + } + + /** + * Replay a schema add/rename/drop via `ALTER TABLE` (mirroring the capture-side ops), so the + * engine enforces column-mapping requirements — e.g. it rejects a rename/drop on a non-CM table + * that would reinterpret physical columns by the new logical name. Each `EvolveSchemaCommit` + * carries exactly one of add/rename/drop, so this emits exactly one commit. + */ + private def replayEvolveSchema( + spark: SparkSession, commit: EvolveSchemaCommit, tableRef: String): Unit = { + commit.addColumns.foreach { cols => + val st = StructType(cols.asInstanceOf[Seq[Map[String, Any]]].map { col => + StructField( + col("name").toString, + DataType.fromJson(JsonUtil.mapper.writeValueAsString(col("type"))), + col.get("nullable").forall(_.asInstanceOf[Boolean])) + }) + spark.sql(s"ALTER TABLE $tableRef ADD COLUMNS (${st.toDDL})") + } + commit.renameColumns.foreach { renames => + for ((oldName, newName) <- renames) + spark.sql(s"ALTER TABLE $tableRef RENAME COLUMN $oldName TO $newName") + } + commit.dropColumns.foreach { drops => + if (drops.size == 1) spark.sql(s"ALTER TABLE $tableRef DROP COLUMN ${drops.head}") + else spark.sql(s"ALTER TABLE $tableRef DROP COLUMNS (${drops.mkString(", ")})") + } + } + + /** + * Replay a low-level commit: write each add's bundled logical Parquet through the engine + * (via `req.addDataParquet`, so column-mapping/partition/stats are engine-handled), and tombstone + * each remove's referenced prior add, resolved to this replay table's own paths at that version. + */ + private def replayLowLevelCommit( + spark: SparkSession, commit: LowLevelCommitOp, idx: Int, + tablePath: String, testDir: Path): Unit = { + val addDataParquet = commit.addFiles.getOrElse(Seq.empty) + .map(af => testDir.resolve(af.dataFile).toAbsolutePath.toString) + val removePaths = commit.removeFiles.getOrElse(Seq.empty) + .flatMap(rf => SpecLayout.addPathsAt(java.nio.file.Paths.get(tablePath), rf.addedAtCommit)) + DeltaHarness.get.commit(spark, tablePath, + CommitRequest( + addDataParquet = addDataParquet, + schemaJson = commit.schema.map(s => structOf(s).json), + properties = commit.tableProperties, + setTransaction = commit.txn, + removeFiles = removePaths.map(p => CommitRemoveFile(p, dataChange = true)), + addDomainMetadata = commit.addDomainMetadata.getOrElse(Seq.empty), + removeDomainMetadata = commit.removeDomainMetadata.getOrElse(Seq.empty))) + } + + /** Read the bundled Parquet (full rows, incl. partition columns) and Append it. */ + private def loadDataFiles( + spark: SparkSession, files: Seq[String], tablePath: String, testDir: Path): Unit = { + val resolved = files.map(testDir.resolve).filter(Files.exists(_)).map(_.toAbsolutePath.toString) + if (resolved.nonEmpty) { + spark.read.parquet(resolved: _*).write.format("delta").mode(SaveMode.Append).save(tablePath) + } + } + + /** + * Reconstruct a Spark [[StructType]] from a stored Delta schema value. The spec stores schemas + * as `StructType.json` deserialized to a Map (see `WriteSpecCapture.ddlToSchemaJson`), so this + * re-serializes and parses it through Spark's own reader. `.toDDL`/`.sql`/`.json` on the result + * are then exact and lossless (decimal precision, nested nullability, quoting all preserved). + */ + private def structOf(schema: Any): StructType = + DataType.fromJson(JsonUtil.mapper.writeValueAsString(schema)).asInstanceOf[StructType] } diff --git a/workload-generator/src/main/scala/io/delta/workload/WriteSpecCapture.scala b/workload-generator/src/main/scala/io/delta/workload/WriteSpecCapture.scala new file mode 100644 index 00000000..6f04c2d8 --- /dev/null +++ b/workload-generator/src/main/scala/io/delta/workload/WriteSpecCapture.scala @@ -0,0 +1,158 @@ +/* + * Copyright (2025) The Delta Lake Project Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.delta.workload + +import java.nio.file.{Files, Path} + +import scala.collection.mutable + +import org.apache.spark.sql.{SaveMode, SparkSession} +import org.apache.spark.sql.types.StructType + +import io.delta.workload.deltaharness.DeltaHarness + +/** + * Records structured write operations and serializes them as a `write` spec + * (`specs/_write.json`), the portable recipe a Delta writer replays to reconstruct a table. + * + * High-level operations carry SQL semantics (create_table, insert, update, ...); the low-level + * `commit` carries raw Delta actions. Data-bearing ops (insert, replace-as-select, low-level adds) + * record their LOGICAL rows, which `buildSpec` materializes to Parquet under `data/` so the spec + * is self-sufficient for replay. + */ +class WriteSpecBuilder { + + /** + * A recorded commit together with its capture-time logical rows. Rows are NOT part of the + * serialized [[WriteCommit]] (which carries only file paths); [[buildSpec]] materializes them to + * Parquet and writes the resulting paths back into the commit. `rows` holds one row-set per data + * file the commit adds: high-level insert / replace-as-select add a single file (one element); + * a low-level commit adds one row-set per `AddFile`. Data-free ops carry no rows. + */ + private case class WriteOpWithData(commit: WriteCommit, rows: Seq[Seq[Map[String, Any]]]) + + private val recorded = mutable.ArrayBuffer[WriteOpWithData]() + + /** The ordinal (== table version) the next recorded commit will occupy. */ + def nextOrdinal: Int = recorded.size + + /** + * Append a recorded commit and return its ordinal (== the table version it produces). Pass the + * commit's logical `rows` (one row-set per data file it adds) to materialize to Parquet in + * [[buildSpec]]; data-free ops pass nothing. + */ + def record(commit: WriteCommit, rows: Seq[Seq[Map[String, Any]]] = Nil): Int = { + recorded += WriteOpWithData(commit, rows) + recorded.size - 1 + } + + /** Parse `ADD COLUMNS` DDL into the spec's `addColumns` JSON (name/type/nullable per field). */ + private[workload] def addColumnsJson(ddl: String): Option[Any] = + if (ddl.nonEmpty) { + Some(StructType.fromDDL(ddl).fields.map { f => + val typeJson = JsonUtil.mapper.readValue(f.dataType.json, classOf[Any]) + Map[String, Any]("name" -> f.name, "type" -> typeJson, "nullable" -> f.nullable) + }.toSeq) + } else None + + /** Parse a SQL DDL string into Delta schema JSON using Spark's type parser. */ + private[workload] def ddlToSchemaJson(ddl: String): Any = { + val st = StructType.fromDDL(ddl) + JsonUtil.mapper.readValue(st.json, classOf[Any]) + } + + // === Serialization === + + /** + * Serialize recorded operations to `specs/_write.json`, enriching the data-producing + * commits (create_table, insert) with the actual data files they added, and writing + * the expected final table state to `expected/_write/`. + * + * New data files are copied to `outputDir/data/commit_/` and referenced via + * relative paths in each commit's `dataFiles`, making the spec self-sufficient. + */ + def buildSpec(spark: SparkSession, tablePath: Path, outputDir: Path): Unit = { + val log = DeltaHarness.get.openLog(spark, tablePath.toString) + + // Commit index lines up with the table version it produced (commit 0 is the create at + // version 0). Guard the coupling: each recorded op must map to exactly one version. + val finalVersion = log.update().version + require(recorded.size == finalVersion + 1, + s"write spec records ${recorded.size} ops but table has ${finalVersion + 1} versions " + + "(0.." + finalVersion + "); op index can no longer be used as the table version") + + // Materialize the spec's data files. The API took `rows`; the spec stores Parquet. High-level + // insert/replace-as-select rows are written to `data/commit_/part-00000.parquet` (full + // schema incl. partition columns, so replay can Append/RTAS). Low-level commits copy each + // add's already-written Parquet to `data/commit_/` (same in-table path). + val enriched = recorded.zipWithIndex.map { + case (WriteOpWithData(ins: InsertCommit, rows), idx) => + ins.copy(dataFiles = materializeRows(spark, tablePath.toString, outputDir, idx, rows)) + case (WriteOpWithData(rep: ReplaceTableCommit, rows), idx) => + rep.copy(dataFiles = materializeRows(spark, tablePath.toString, outputDir, idx, rows)) + case (WriteOpWithData(ll: LowLevelCommitOp, rows), idx) => + // Materialize each low-level add's logical rows to the `dataFile` path recorded in the + // action (full schema incl. partition columns, so replay's writeFiles can partition/map). + if (rows.nonEmpty) { + val schema = DeltaHarness.get.schemaAt(spark, tablePath.toString, Some(idx.toLong)) + ll.addFiles.getOrElse(Seq.empty).zip(rows).foreach { case (af, r) => + DeltaHarness.get.writeRows(spark, schema, r, outputDir.resolve(af.dataFile)) + } + } + ll + case (WriteOpWithData(other, _), _) => other + } + + // `write` is just another spec: it lives in `specs/_write.json` alongside read/snapshot + // specs, with its final-state under `expected/_write/`. + val writeName = s"${outputDir.getFileName.toString}_write" + JsonUtil.writeSpec( + outputDir.resolve("specs").resolve(s"$writeName.json"), WriteSpec(enriched.toSeq)) + writeExpectedLatest(spark, tablePath, outputDir.resolve("expected").resolve(writeName)) + } + + /** + * Write the rows recorded for commit `idx` (if any) to `data/commit_/part-00000.parquet` + * using the table's schema at that version, and return the spec-relative path. None if the op + * recorded no rows. + */ + private def materializeRows( + spark: SparkSession, tablePath: String, outputDir: Path, idx: Int, + rows: Seq[Seq[Map[String, Any]]]): Option[Seq[String]] = { + rows.headOption.filter(_.nonEmpty).map { r => + val schema = DeltaHarness.get.schemaAt(spark, tablePath, Some(idx.toLong)) + val rel = SpecLayout.commitDataFile(idx, "part-00000.parquet") + DeltaHarness.get.writeRows(spark, schema, r, outputDir.resolve(rel)) + Seq(rel) + } + } + + /** + * Capture the write spec's expected final rows under `expected/_write/table_content/`, + * which [[WorkloadValidator]] compares against the replayed table. + */ + private def writeExpectedLatest(spark: SparkSession, tablePath: Path, latestDir: Path): Unit = { + Files.createDirectories(latestDir) + + val contentDir = latestDir.resolve("table_content") + if (contentDir.toFile.exists()) { + org.apache.commons.io.FileUtils.deleteDirectory(contentDir.toFile) + } + spark.read.format("delta").load(tablePath.toString) + .write.mode(SaveMode.Overwrite).parquet(contentDir.toString) + } +} diff --git a/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaHarness.scala b/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaHarness.scala index 87fae79a..3408da0b 100644 --- a/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaHarness.scala +++ b/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaHarness.scala @@ -16,7 +16,42 @@ package io.delta.workload.deltaharness +import java.nio.file.Path + import org.apache.spark.sql.{DataFrame, SparkSession} +import org.apache.spark.sql.types.StructType + +import io.delta.workload.{AddDomainMetadata, AppTxn} + +/** + * A remove-file action (tombstone) for a low-level commit, by in-table `path`. The implementation + * tombstones the matching active file via the engine, so the tombstone inherits its + * `partitionValues`/`size`/`stats` (column-mapping- and partition-correct) — the caller supplies + * only the path and the `dataChange` flag. + */ +case class CommitRemoveFile(path: String, dataChange: Boolean) + +/** + * A platform-neutral description of a low-level commit. Carries only plain Scala types and the + * generator's neutral action case classes, never `org.apache.spark.sql.delta.*`. + * + * @param addDataParquet Parquet files of LOGICAL rows to add; the engine reads each and writes + * it into the table (column-mapping aware, computing stats/layout) + * @param schemaJson `StructType.json` to set as `metadata.schemaString`, or None to keep + * @param properties properties merged into `metadata.configuration`, or None to keep + * @param setTransaction optional `SetTransaction` (appId/version) action + * @param removeFiles remove-file actions to tombstone (by in-table path) + * @param addDomainMetadata domain-metadata entries to add + * @param removeDomainMetadata domain names to tombstone + */ +case class CommitRequest( + addDataParquet: Seq[String] = Nil, + schemaJson: Option[String] = None, + properties: Option[Map[String, String]] = None, + setTransaction: Option[AppTxn] = None, + removeFiles: Seq[CommitRemoveFile] = Nil, + addDomainMetadata: Seq[AddDomainMetadata] = Nil, + removeDomainMetadata: Seq[String] = Nil) /** * Platform-specific backing for Delta-internal access. @@ -33,6 +68,31 @@ trait DeltaHarness { * cache management an adapter-internal concern. */ def openLog(spark: SparkSession, tablePath: String): LogView + + /** + * Open a transaction at `tablePath`, apply the metadata update (schema/properties) if present, + * write any `req.addDataParquet` data files through the engine's own write path (honoring column + * mapping and partitioning, computing stats), and commit the produced AddFile actions together + * with `req`'s other actions in one `DeltaOperations.ManualUpdate`. + * + * Implementations MUST clear any internal DeltaLog cache before and after committing, mirroring + * the `openLog` cache-clearing contract so consumers always see a fresh view. + * + * @return the in-table `AddFile.path`s the engine produced (in `addDataParquet` order; a file + * may yield several adds when partitioned), so callers can reference them later. + */ + def commit(spark: SparkSession, tablePath: String, req: CommitRequest): Seq[String] + + /** The table's schema at `version` (latest if None). */ + def schemaAt(spark: SparkSession, tablePath: String, version: Option[Long]): StructType + + /** + * Materialize in-memory `rows` (the workload-generator's authoring surface) into a single Parquet + * file at `dest`, coercing each value to the corresponding column type in `schema`. Capture writes + * rows here at generation time — no Delta-log scan — and replay reads the produced files back. + */ + def writeRows( + spark: SparkSession, schema: StructType, rows: Seq[Map[String, Any]], dest: Path): Unit } trait LogView { diff --git a/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaSparkHarness.scala b/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaSparkHarness.scala index 108befc6..0e40cd30 100644 --- a/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaSparkHarness.scala +++ b/workload-generator/src/main/scala/io/delta/workload/deltaharness/DeltaSparkHarness.scala @@ -16,14 +16,150 @@ package io.delta.workload.deltaharness -import org.apache.spark.sql.{DataFrame, SparkSession} -import org.apache.spark.sql.delta.{DeltaLog, Snapshot} +import java.nio.file.{Files, Path, StandardCopyOption} + +import scala.collection.mutable +import scala.jdk.CollectionConverters._ + +import org.apache.commons.io.FileUtils +import org.apache.spark.sql.{DataFrame, Row, SparkSession} +import org.apache.spark.sql.delta.{DeltaLog, DeltaOperations, Snapshot} +import org.apache.spark.sql.delta.actions.{Action, AddFile, DomainMetadata, FileAction, SetTransaction} +import org.apache.spark.sql.types._ + +import io.delta.workload.JsonUtil class DeltaSparkHarness extends DeltaHarness { override def openLog(spark: SparkSession, tablePath: String): LogView = { DeltaLog.clearCache() new DeltaSparkLogView(DeltaLog.forTable(spark, tablePath)) } + + override def commit(spark: SparkSession, tablePath: String, req: CommitRequest): Seq[String] = { + DeltaLog.clearCache() + val deltaLog = DeltaLog.forTable(spark, tablePath) + val txn = deltaLog.startTransaction() + val now = System.currentTimeMillis() + + // Metadata (schema/properties) is updated first so writeFiles writes against the new schema. + if (req.schemaJson.isDefined || req.properties.isDefined) { + val current = txn.metadata + txn.updateMetadata(current.copy( + schemaString = req.schemaJson.getOrElse(current.schemaString), + configuration = req.properties.map(current.configuration ++ _).getOrElse(current.configuration))) + } + + // Engine-written data files: column mapping / partitioning / stats all handled by writeFiles. + // Commit ALL file actions it returns (AddFile + AddCDCFile when change data feed is enabled); + // only the AddFile paths are returned (for ordinal-based remove resolution). + val fileActions: Seq[FileAction] = req.addDataParquet.flatMap { p => + txn.writeFiles(spark.read.parquet(p)) + } + + val manual = mutable.ArrayBuffer[Action]() + req.setTransaction.foreach { t => manual += SetTransaction(t.appId, t.version, Some(now)) } + + if (req.removeFiles.nonEmpty) { + // Tombstone the matching ACTIVE file so the RemoveFile inherits its partitionValues/size/ + // stats (extendedFileMetadata) — column-mapping- and partition-correct, derived not guessed. + val active = txn.snapshot.allFiles.collect().map(a => a.path -> a).toMap + req.removeFiles.foreach { rf => + val add = active.getOrElse(rf.path, throw new IllegalStateException( + s"removeFiles references a path not active in the table: ${rf.path}")) + manual += add.removeWithTimestamp(now, rf.dataChange) + } + } + + req.addDomainMetadata.foreach { dm => manual += DomainMetadata(dm.domain, dm.configuration, removed = false) } + req.removeDomainMetadata.foreach { domain => manual += DomainMetadata(domain, "", removed = true) } + + txn.commit(fileActions ++ manual.toSeq, DeltaOperations.ManualUpdate) + DeltaLog.clearCache() + fileActions.collect { case a: AddFile => a.path } + } + + override def schemaAt( + spark: SparkSession, tablePath: String, version: Option[Long]): StructType = { + val log = openLog(spark, tablePath) + val snapshot = version.map(log.getSnapshotAt).getOrElse(log.update()) + val meta = JsonUtil.mapper.readTree(snapshot.metadataJson).get("metaData") + DataType.fromJson(meta.get("schemaString").asText()).asInstanceOf[StructType] + } + + override def writeRows( + spark: SparkSession, schema: StructType, rows: Seq[Map[String, Any]], dest: Path): Unit = { + val sparkRows = rows.map { r => + Row.fromSeq(schema.fields.map(f => coerce(r.get(f.name), f.dataType)).toSeq) + } + val tmp = Files.createTempDirectory("row-parquet") + try { + spark.createDataFrame(sparkRows.asJava, schema) + .coalesce(1).write.mode("overwrite").parquet(tmp.toString) + val part = Files.list(tmp).iterator().asScala.find { p => + val n = p.getFileName.toString + n.endsWith(".parquet") && !n.startsWith(".") + }.getOrElse(throw new IllegalStateException(s"no parquet part-file written under $tmp")) + Files.createDirectories(dest.getParent) + Files.move(part, dest, StandardCopyOption.REPLACE_EXISTING) + } finally { + try FileUtils.deleteDirectory(tmp.toFile) catch { case _: Throwable => } + } + } + + /** + * Coerce a row's value to the column's Spark type. `None`/`null` -> SQL NULL. Unknown types + * fail loud rather than passing the raw value through: a silent passthrough that Spark happens + * to accept could write a wrong-but-symmetric value (wrong on both capture AND replay), which + * the row comparison would not catch — a false pass. + */ + private def coerce(value: Option[Any], dataType: DataType): Any = + value.flatMap(Option(_)) match { + case None => null + case Some(v) => dataType match { + case _: IntegerType => v.asInstanceOf[Number].intValue() + case _: LongType => v.asInstanceOf[Number].longValue() + case _: ShortType => v.asInstanceOf[Number].shortValue() + case _: ByteType => v.asInstanceOf[Number].byteValue() + case _: DoubleType => v.asInstanceOf[Number].doubleValue() + case _: FloatType => v.asInstanceOf[Number].floatValue() + case _: BooleanType => v.asInstanceOf[Boolean] + case _: StringType => v.toString + case _: DateType => v match { + case s: String => java.sql.Date.valueOf(s) + case d: java.sql.Date => d + case n: Number => new java.sql.Date(n.longValue()) + case _ => unsupported(v, dataType) + } + case _: TimestampType => v match { + case s: String => java.sql.Timestamp.valueOf(s) + case t: java.sql.Timestamp => t + case n: Number => new java.sql.Timestamp(n.longValue()) + case _ => unsupported(v, dataType) + } + case _: TimestampNTZType => v match { + case s: String => java.time.LocalDateTime.parse(s.replace(' ', 'T')) + case ldt: java.time.LocalDateTime => ldt + case _ => unsupported(v, dataType) + } + case _: DecimalType => v match { + case bd: java.math.BigDecimal => bd + case bd: scala.math.BigDecimal => bd.bigDecimal + case s: String => new java.math.BigDecimal(s) + case n: Number => new java.math.BigDecimal(n.toString) + case _ => unsupported(v, dataType) + } + case _: BinaryType => v match { + case b: Array[Byte] => b + case s: String => java.util.Base64.getDecoder.decode(s) + case _ => unsupported(v, dataType) + } + case _ => unsupported(v, dataType) + } + } + + private def unsupported(v: Any, dataType: DataType): Nothing = + throw new IllegalArgumentException( + s"coerce: unsupported value '$v' (${v.getClass.getName}) for column type $dataType") } private class DeltaSparkLogView(inner: DeltaLog) extends LogView { diff --git a/workload-generator/src/main/scala/io/delta/workload/log/CommitLog.scala b/workload-generator/src/main/scala/io/delta/workload/log/CommitLog.scala index 38d90d25..d1be12dc 100644 --- a/workload-generator/src/main/scala/io/delta/workload/log/CommitLog.scala +++ b/workload-generator/src/main/scala/io/delta/workload/log/CommitLog.scala @@ -60,7 +60,13 @@ object CommitLog { def write(tableDir: Path, version: Long, actions: Seq[Action]): Unit = writeRaw(tableDir, version, actions.map(_.toJson)) - /** Read → transform → write. The common case for corruption / mutation tests. */ + /** + * Read → transform → write. Actions are re-serialized from the typed model, so this is for + * action-level edits (filter, reorder, `.copy` fields); it does not preserve fields the ADT + * does not model or the exact byte layout of untouched lines. For byte-exact corruption, write + * the raw lines directly (e.g. via mutateTable). Unrecognized/malformed lines round-trip as + * [[RawAction]], so they are preserved verbatim. + */ def mutate(tableDir: Path, version: Long)(fn: Seq[Action] => Seq[Action]): Unit = write(tableDir, version, fn(read(tableDir, version))) } diff --git a/workload-generator/src/test/scala/io/delta/workload/WorkloadGeneratorSuite.scala b/workload-generator/src/test/scala/io/delta/workload/WorkloadGeneratorSuite.scala index 2d34cff4..d0d9f523 100644 --- a/workload-generator/src/test/scala/io/delta/workload/WorkloadGeneratorSuite.scala +++ b/workload-generator/src/test/scala/io/delta/workload/WorkloadGeneratorSuite.scala @@ -25,6 +25,8 @@ import org.apache.spark.sql.SparkSession import org.scalatest.BeforeAndAfterAll import org.scalatest.funsuite.AnyFunSuite +import io.delta.workload.log.{AddFile, Metadata} + class WorkloadGeneratorSuite extends AnyFunSuite with BeforeAndAfterAll with WorkloadOps { private var _spark: SparkSession = _ @@ -609,8 +611,8 @@ class WorkloadGeneratorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor sql("INSERT INTO tbl VALUES (1),(2)") val t = registerTable("tbl") modifyCommitActions(t, version = 1) { actions => - actions.map { case ("add", node) => - node.put("stats", """{"numRecords":999}"""); ("add", node) + actions.map { + case a: AddFile => a.copy(stats = Some("""{"numRecords":999}""")) case other => other } } @@ -639,7 +641,7 @@ class WorkloadGeneratorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor sql("INSERT INTO tbl VALUES (1),(2)") val t = registerTable("tbl") modifyCommitActions(t, version = 1) { actions => - actions.filter(_._1 != "add") + actions.filterNot(_.isInstanceOf[AddFile]) } snapshotSpec(t) } @@ -660,11 +662,8 @@ class WorkloadGeneratorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor val t = registerTable("tbl") // Modify the metaData action in the CREATE commit (v0) modifyCommitActions(t, version = 0) { actions => - actions.map { case ("metaData", node) => - val configNode = node.get("configuration").asInstanceOf[ - com.fasterxml.jackson.databind.node.ObjectNode] - configNode.put("test.property", "hello") - ("metaData", node) + actions.map { + case m: Metadata => m.copy(configuration = m.configuration + ("test.property" -> "hello")) case other => other } } @@ -753,7 +752,7 @@ class WorkloadGeneratorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor test("framework: zero-table test warns") { val results = run("t_fw3") { _ => sql("CREATE TABLE tbl (id INT) USING delta") - // Never call registerTable() — should warn + // Never call registerTable(); should warn } assert(results.size == 1) assert(results.head.passed) // passes but with warning diff --git a/workload-generator/src/test/scala/io/delta/workload/WorkloadValidatorSuite.scala b/workload-generator/src/test/scala/io/delta/workload/WorkloadValidatorSuite.scala index 74f4c809..8864730b 100644 --- a/workload-generator/src/test/scala/io/delta/workload/WorkloadValidatorSuite.scala +++ b/workload-generator/src/test/scala/io/delta/workload/WorkloadValidatorSuite.scala @@ -96,6 +96,26 @@ class WorkloadValidatorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor s"expected >= 2 specs validated, got ${result.passed}") } + test("validates a write workload (write-derived dispatch + replay)") { + spark.sql("DROP TABLE IF EXISTS wtbl") + generate("val_write") { + val w = createTableOp("wtbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + // Validate the write workload standalone (post-generation): the write spec replays into a + // fresh table and the write-derived read/snapshot specs validate against it. + val result = WorkloadValidator.validateTestDir(spark, outputDir.resolve("val_write")) + assert(result.success, + s"write workload should validate; errors:\n ${result.errors.mkString("\n ")}") + assert(result.passed >= 3, + s"expected >= 3 specs (write + read + snapshot), got ${result.passed}") + } + test("validator reports failures when the table diverges from the spec") { spark.sql("DROP TABLE IF EXISTS tbl2") generate("val_divergence") { diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/CorruptionSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/CorruptionSuite.scala index d6b056e5..c0fba1db 100644 --- a/workload-generator/src/test/scala/io/delta/workload/tables/CorruptionSuite.scala +++ b/workload-generator/src/test/scala/io/delta/workload/tables/CorruptionSuite.scala @@ -17,6 +17,7 @@ package io.delta.workload.tables import io.delta.workload.WorkloadTestSuite +import io.delta.workload.log.AddFile class CorruptionSuite extends WorkloadTestSuite("corruption") { @@ -80,8 +81,8 @@ class CorruptionSuite extends WorkloadTestSuite("corruption") { sql("INSERT INTO tbl VALUES (1,'a'),(2,'b'),(3,'c')") val t = registerTable("tbl") modifyCommitActions(t, version = 1) { actions => - actions.map { case ("add", node) => - node.put("stats", """{"numRecords":999}"""); ("add", node) + actions.map { + case a: AddFile => a.copy(stats = Some("""{"numRecords":999}""")) case other => other } } diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/DataSkippingSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/DataSkippingSuite.scala index 0fb987d9..dbc96d90 100644 --- a/workload-generator/src/test/scala/io/delta/workload/tables/DataSkippingSuite.scala +++ b/workload-generator/src/test/scala/io/delta/workload/tables/DataSkippingSuite.scala @@ -16,7 +16,8 @@ package io.delta.workload.tables -import io.delta.workload.WorkloadTestSuite +import io.delta.workload.{JsonUtil, WorkloadTestSuite} +import io.delta.workload.log.AddFile /** * Data skipping, statistics, and partitioning workloads. @@ -880,8 +881,8 @@ class DataSkippingSuite extends WorkloadTestSuite("data_skipping") { val t = registerTable("tbl") // Strip stats from commit modifyCommitActions(t, version = 1) { actions => - actions.map { case ("add", node) => - node.remove("stats"); ("add", node) + actions.map { + case a: AddFile => a.copy(stats = None) case other => other } } @@ -1190,19 +1191,13 @@ class DataSkippingSuite extends WorkloadTestSuite("data_skipping") { val t = registerTable("tbl") // Strip min/max stats, keep only numRecords modifyCommitActions(t, 0) { actions => - actions.map { case ("add", node) => - if (node.has("stats")) { - val stats = node.get("stats").asText() - if (stats.contains("numRecords")) { - import com.fasterxml.jackson.databind.ObjectMapper - val mapper = new ObjectMapper() - val statsNode = mapper.readTree(stats) - val newStats = mapper.createObjectNode() - newStats.set("numRecords", statsNode.get("numRecords")) - node.put("stats", mapper.writeValueAsString(newStats)) - } - } - ("add", node) + actions.map { + case a: AddFile if a.stats.exists(_.contains("numRecords")) => + val statsNode = JsonUtil.mapper.readTree(a.stats.get) + val newStats = JsonUtil.mapper.createObjectNode() + newStats.set[com.fasterxml.jackson.databind.JsonNode]( + "numRecords", statsNode.get("numRecords")) + a.copy(stats = Some(JsonUtil.mapper.writeValueAsString(newStats))) case other => other } } @@ -1253,9 +1248,8 @@ class DataSkippingSuite extends WorkloadTestSuite("data_skipping") { val t = registerTable("tbl") // Strip stats completely to simulate empty stats string modifyCommitActions(t, 0) { actions => - actions.map { case ("add", node) => - if (node.has("stats")) node.put("stats", "") - ("add", node) + actions.map { + case a: AddFile if a.stats.isDefined => a.copy(stats = Some("")) case other => other } } @@ -1269,8 +1263,8 @@ class DataSkippingSuite extends WorkloadTestSuite("data_skipping") { val t = registerTable("tbl") // Remove stats field entirely modifyCommitActions(t, 0) { actions => - actions.map { case ("add", node) => - node.remove("stats"); ("add", node) + actions.map { + case a: AddFile => a.copy(stats = None) case other => other } } diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/WriteBasicSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/WriteBasicSuite.scala new file mode 100644 index 00000000..d969570d --- /dev/null +++ b/workload-generator/src/test/scala/io/delta/workload/tables/WriteBasicSuite.scala @@ -0,0 +1,243 @@ +/* + * Copyright (2025) The Delta Lake Project Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.delta.workload.tables + +import io.delta.workload.WorkloadTestSuite + +/** Basic write workloads producing a write spec (specs/<name>_write.json) for Delta writer testing. */ +class WriteBasicSuite extends WorkloadTestSuite("write_basic") { + + test("create_and_read") { + val w = createTableOp("tbl", schema = "id INT, name STRING, score DOUBLE") + insertOp(w, Seq(("alice", 95.5), ("bob", 87.3), ("charlie", 72.1)).zipWithIndex.map { + case ((name, score), i) => Map("id" -> (i + 1), "name" -> name, "score" -> score) }) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "score > 90.0", name = "read_high_score") + snapshotSpec(t) + } + + test("create_with_properties") { + val w = createTableOp("tbl", + schema = "id INT, value STRING, amount INT", + properties = Map( + "delta.enableDeletionVectors" -> "true", + "delta.enableChangeDataFeed" -> "true")) + insertOp(w, Seq("alpha", "beta", "gamma").zipWithIndex.map { + case (v, i) => Map("id" -> (i + 1), "value" -> v, "amount" -> (i + 1) * 100) }) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("insert_multiple") { + val w = createTableOp("tbl", schema = "id INT, category STRING") + insertOp(w, (1 to 2).map(i => Map("id" -> i, "category" -> "a"))) + insertOp(w, (3 to 5).map(i => Map("id" -> i, "category" -> "b"))) + insertOp(w, Seq(Map("id" -> 6, "category" -> "c"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_v1") + readSpec(t, version = 2, name = "read_v2") + readSpec(t, version = 3, name = "read_v3") + readSpec(t, predicate = "category = 'b'", name = "read_category_b") + snapshotSpec(t) + } + + test("delete_basic") { + val w = createTableOp("tbl", + schema = "id INT, value STRING, amount INT", + properties = Map("delta.enableDeletionVectors" -> "true")) + insertOp(w, (1 to 5).map(i => + Map("id" -> i, "value" -> (if (i % 2 == 0) "remove" else "keep"), "amount" -> i * 10))) + deleteOp(w, predicate = "value = 'remove'") + val t = registerWriteSpec(w) + readSpec(t, name = "read_after_delete") + readSpec(t, version = 1, name = "read_before_delete") + readSpec(t, predicate = "amount > 25", name = "read_large_amount") + snapshotSpec(t) + } + + test("update_basic") { + val w = createTableOp("tbl", + schema = "id INT, status STRING, count INT", + properties = Map("delta.enableDeletionVectors" -> "true")) + insertOp(w, Seq(("pending", 0), ("pending", 0), ("active", 5), ("pending", 0)).zipWithIndex.map { + case ((status, count), i) => Map("id" -> (i + 1), "status" -> status, "count" -> count) }) + updateOp(w, predicate = "status = 'pending'", + set = Map("status" -> "'active'", "count" -> "count + 1")) + val t = registerWriteSpec(w) + readSpec(t, name = "read_after_update") + readSpec(t, version = 1, name = "read_before_update") + readSpec(t, predicate = "status = 'active'", name = "read_active") + snapshotSpec(t) + } + + test("alter_add_column") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq("alice", "bob").zipWithIndex.map { case (n, i) => Map("id" -> (i + 1), "name" -> n) }) + addColumnsOp(w, "email STRING") + insertOp(w, Seq("charlie", "diana").zipWithIndex.map { + case (n, i) => Map("id" -> (i + 3), "name" -> n, "email" -> s"$n@test.com") }) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_before_alter") + readSpec(t, predicate = "email IS NULL", name = "read_null_email") + readSpec(t, predicate = "email IS NOT NULL", name = "read_with_email") + snapshotSpec(t) + } + + test("alter_set_properties") { + val w = createTableOp("tbl", schema = "id INT, value INT") + insertOp(w, (1 to 3).map(i => Map("id" -> i, "value" -> i * 100))) + setPropertiesOp(w, Map("delta.enableChangeDataFeed" -> "true")) + insertOp(w, (4 to 5).map(i => Map("id" -> i, "value" -> i * 100))) + updateOp(w, predicate = "id <= 2", set = Map("value" -> "value + 1000")) + val t = registerWriteSpec(w) + readSpec(t, version = 1, name = "read_initial_insert") + readSpec(t, version = 3, name = "read_before_update") + readSpec(t, name = "read_all") + readSpec(t, predicate = "value > 1000", name = "read_updated") + snapshotSpec(t) + } + + test("delete_all_rows") { + val w = createTableOp("tbl", + schema = "id INT, data STRING", + properties = Map("delta.enableDeletionVectors" -> "true")) + insertOp(w, Seq("first", "second", "third", "fourth").zipWithIndex.map { + case (d, i) => Map("id" -> (i + 1), "data" -> d) }) + deleteOp(w, predicate = "true") + val t = registerWriteSpec(w) + readSpec(t, name = "read_after_delete_all") + readSpec(t, version = 1, name = "read_before_delete_all") + snapshotSpec(t) + } + + test("partitioned_insert") { + val w = createTableOp("tbl", + schema = "id INT, region STRING, revenue INT", + partitionColumns = Seq("region")) + insertOp(w, Seq(100, 150).zipWithIndex.map { + case (rev, i) => Map("id" -> (i + 1), "region" -> "east", "revenue" -> rev) }) + insertOp(w, Seq(200, 250, 300).zipWithIndex.map { + case (rev, i) => Map("id" -> (i + 3), "region" -> "west", "revenue" -> rev) }) + insertOp(w, Seq(Map("id" -> 6, "region" -> "north", "revenue" -> 400))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "region = 'east'", name = "read_east") + readSpec(t, predicate = "region = 'west'", name = "read_west") + readSpec(t, predicate = "region = 'north'", name = "read_north") + readSpec(t, predicate = "revenue >= 200", name = "read_high_revenue") + snapshotSpec(t) + } + + test("replace_table_as_select") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq("alice", "bob").zipWithIndex.map { case (n, i) => Map("id" -> (i + 1), "name" -> n) }) + // Replace-as-select: new schema + new data in a single commit. + replaceTableOp(w, + schema = "id INT, label STRING, score DOUBLE", + rows = Seq(("x", 1.5), ("y", 2.5)).zipWithIndex.map { + case ((label, score), i) => Map("id" -> ((i + 1) * 10), "label" -> label, "score" -> score) }) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_before_replace") + readSpec(t, predicate = "score > 2.0", name = "read_high_score") + snapshotSpec(t) + } + + test("replace_table_schema_only") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // Pure replace (no data) then insert under the new schema. + replaceTableOp(w, + schema = "a BIGINT, b STRING", + properties = Map("delta.enableChangeDataFeed" -> "true")) + insertOp(w, Seq(Map("a" -> 100L, "b" -> "hello"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("replace_table_as_select_partitioned") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + replaceTableOp(w, + schema = "id INT, region STRING, revenue INT", + partitionColumns = Seq("region"), + rows = Seq("east", "west", "east").zipWithIndex.map { + case (region, i) => Map("id" -> ((i + 1) * 10), "region" -> region, "revenue" -> ((i + 1) * 100)) }) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "region = 'east'", name = "read_east") + readSpec(t, predicate = "revenue > 150", name = "read_high") + snapshotSpec(t) + } + + test("alter_rename_column") { + val w = createTableOp("tbl", schema = "id INT, name STRING", + properties = Map("delta.columnMapping.mode" -> "name")) + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"), Map("id" -> 2, "name" -> "bob"))) + renameColumnOp(w, "name", "full_name") + insertOp(w, Seq(Map("id" -> 3, "full_name" -> "charlie"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_before_rename") + readSpec(t, predicate = "full_name = 'alice'", name = "read_by_renamed") + snapshotSpec(t) + } + + test("alter_drop_column") { + val w = createTableOp("tbl", schema = "id INT, name STRING, scratch STRING", + properties = Map("delta.columnMapping.mode" -> "name")) + insertOp(w, Seq(Map("id" -> 1, "name" -> "a", "scratch" -> "x"))) + dropColumnsOp(w, Seq("scratch")) + insertOp(w, Seq(Map("id" -> 2, "name" -> "b"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_before_drop") + snapshotSpec(t) + } + + test("alter_unset_properties") { + val w = createTableOp("tbl", schema = "id INT, value INT", + properties = Map("delta.enableChangeDataFeed" -> "true")) + insertOp(w, Seq(Map("id" -> 1, "value" -> 10), Map("id" -> 2, "value" -> 20))) + setPropertiesOp(w, Map("delta.enableDeletionVectors" -> "true")) + unsetPropertiesOp(w, Seq("delta.enableChangeDataFeed")) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("delete_with_dvs") { + val w = createTableOp("tbl", + schema = "id INT, name STRING, score INT", + properties = Map("delta.enableDeletionVectors" -> "true")) + insertOp(w, Seq(("alice", 90), ("bob", 75), ("charlie", 88), ("diana", 92), ("eve", 60), + ("frank", 85)).zipWithIndex.map { + case ((name, score), i) => Map("id" -> (i + 1), "name" -> name, "score" -> score) }) + deleteOp(w, predicate = "score < 80") + val t = registerWriteSpec(w) + readSpec(t, name = "read_after_delete") + readSpec(t, version = 1, name = "read_before_delete") + readSpec(t, predicate = "score >= 85", name = "read_high_score") + snapshotSpec(t) + } + +} diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/WriteCommitSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/WriteCommitSuite.scala new file mode 100644 index 00000000..abfcaa64 --- /dev/null +++ b/workload-generator/src/test/scala/io/delta/workload/tables/WriteCommitSuite.scala @@ -0,0 +1,223 @@ +/* + * Copyright (2025) The Delta Lake Project Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.delta.workload.tables + +import java.nio.file.Files + +import scala.jdk.CollectionConverters._ + +import io.delta.workload.{AddDomainMetadata, AddFileInput, AppTxn, JsonUtil, TableHandle, + WorkloadTestSuite} + +/** + * Low-level `commit` write workloads. Each case drives [[commitOp]] to produce a real `_delta_log` + * entry, asserts the produced action independently of replay, and relies on the generator's + * round-trip validation (replay + re-check specs) for end-to-end coverage. Low-level adds supply + * LOGICAL rows; the engine writes the files (paths/stats/partitionValues are engine-derived). + */ +class WriteCommitSuite extends WorkloadTestSuite("write_commit") { + + /** Action inner-objects of a given type in a commit's `_delta_log/NNN.json`. */ + private def actionsAt(t: TableHandle, version: Long, action: String): Seq[com.fasterxml.jackson.databind.JsonNode] = { + val commitFile = t.sourcePath.resolve("_delta_log").resolve(f"$version%020d.json") + Files.readAllLines(commitFile).asScala.toSeq + .filter(_.trim.nonEmpty) + .map(JsonUtil.mapper.readTree) + .flatMap(n => Option(n.get(action))) + } + + private def latestVersion(t: TableHandle): Long = + io.delta.workload.deltaharness.DeltaHarness.get.openLog(spark, t.sourcePath.toString) + .update().version + + test("commit_with_txn") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, txn = Some(AppTxn("app-1", 7L))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + val txns = actionsAt(t, v, "txn") + assert(txns.size == 1, s"expected one txn action at v$v, got ${txns.size}") + assert(txns.head.get("appId").asText() == "app-1") + assert(txns.head.get("version").asLong() == 7L) + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_with_domain_metadata") { + val w = createTableOp("tbl", + schema = "id INT, name STRING", + properties = Map("delta.feature.domainMetadata" -> "supported")) + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, addDomainMetadata = Some(Seq( + AddDomainMetadata("delta.test", """{"k":"v"}""")))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + val dms = actionsAt(t, v, "domainMetadata") + assert(dms.size == 1, s"expected one domainMetadata action at v$v, got ${dms.size}") + assert(dms.head.get("domain").asText() == "delta.test") + assert(dms.head.get("configuration").asText() == """{"k":"v"}""") + assert(!dms.head.get("removed").asBoolean()) + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_add_file") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // Add a second data file via a low-level commit; the engine writes it and assigns the path. + commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + assert(actionsAt(t, v, "add").size == 1, s"expected one add action at v$v") + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_remove_file") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // Add a file via a low-level commit, then tombstone it by referencing that commit's ordinal. + val added = commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + val addedPath = actionsAt(t, added.toLong, "add").head.get("path").asText() + commitOp(w, removeFiles = Some(Seq(added))) + + val v = latestVersion(t) + val removes = actionsAt(t, v, "remove") + assert(removes.size == 1, s"expected one remove action at v$v, got ${removes.size}") + assert(removes.head.get("path").asText() == addedPath, "remove should tombstone the added file") + + readSpec(t, name = "read_after_remove") + snapshotSpec(t) + } + + test("commit_add_file_with_stats") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // Stats are engine-computed from the rows (numRecords + per-column min/max/nullCount). + commitOp(w, addFiles = Some(Seq(AddFileInput( + rows = Seq(Map("id" -> 2, "name" -> "bob"), Map("id" -> 3, "name" -> "cara")))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + val add = actionsAt(t, v, "add").head + val stats = JsonUtil.mapper.readTree(add.get("stats").asText()) + assert(stats.get("numRecords").asLong() == 2, s"stats: ${add.get("stats").asText()}") + assert(stats.get("minValues").get("id").asInt() == 2) + assert(stats.get("maxValues").get("id").asInt() == 3) + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_partitioned_add") { + val w = createTableOp("tbl", schema = "id INT, region STRING, value INT", + partitionColumns = Seq("region")) + insertOp(w, Seq(Map("id" -> 1, "region" -> "east", "value" -> 10))) + // The rows carry the partition column; the engine partitions and sets partitionValues. + commitOp(w, addFiles = Some(Seq(AddFileInput( + rows = Seq(Map("id" -> 2, "region" -> "west", "value" -> 20)))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + val add = actionsAt(t, v, "add").head + assert(add.get("partitionValues").get("region").asText() == "west") + + readSpec(t, name = "read_all") + readSpec(t, predicate = "region = 'west'", name = "read_west") + snapshotSpec(t) + } + + test("commit_multi_file") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, addFiles = Some(Seq( + AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob"))), + AddFileInput(rows = Seq(Map("id" -> 3, "name" -> "cara")))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + assert(actionsAt(t, v, "add").size == 2, s"expected two add actions at v$v") + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_with_schema_and_properties") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // A single low-level commit that evolves the schema, sets a property, AND adds a file. + commitOp(w, + schemaDDL = Some("id INT, name STRING, extra STRING"), + tableProperties = Some(Map("delta.appendOnly" -> "true")), + addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + val meta = actionsAt(t, v, "metaData").head + assert(meta.get("schemaString").asText().contains("extra"), + s"schema: ${meta.get("schemaString").asText()}") + assert(meta.get("configuration").get("delta.appendOnly").asText() == "true") + + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("commit_remove_multi_file_add") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // One low-level commit adds TWO files; removing by that ordinal must tombstone BOTH. + val added = commitOp(w, addFiles = Some(Seq( + AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob"))), + AddFileInput(rows = Seq(Map("id" -> 3, "name" -> "cara")))))) + val t = registerWriteSpec(w) + val addedPaths = actionsAt(t, added.toLong, "add").map(_.get("path").asText()).toSet + assert(addedPaths.size == 2, s"expected two adds at v$added, got $addedPaths") + commitOp(w, removeFiles = Some(Seq(added))) + + val v = latestVersion(t) + val removed = actionsAt(t, v, "remove").map(_.get("path").asText()).toSet + assert(removed == addedPaths, s"both files tombstoned: removed=$removed added=$addedPaths") + + readSpec(t, name = "read_all") // only id=1 remains + snapshotSpec(t) + } + + test("commit_with_column_mapping") { + // Low-level adds go through the engine write path, so column mapping is handled: the logical + // rows are written under the table's physical names with engine-computed (physical-keyed) stats. + val w = createTableOp("tbl", schema = "id INT, name STRING", + properties = Map("delta.columnMapping.mode" -> "name")) + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + + val v = latestVersion(t) + assert(actionsAt(t, v, "add").size == 1, s"expected one add action at v$v") + + readSpec(t, name = "read_all") + readSpec(t, predicate = "name = 'bob'", name = "read_bob") + snapshotSpec(t) + } +} diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/WriteSequencesSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/WriteSequencesSuite.scala new file mode 100644 index 00000000..3bb953d7 --- /dev/null +++ b/workload-generator/src/test/scala/io/delta/workload/tables/WriteSequencesSuite.scala @@ -0,0 +1,115 @@ +/* + * Copyright (2025) The Delta Lake Project Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.delta.workload.tables + +import io.delta.workload.{AddDomainMetadata, AddFileInput, AppTxn, WorkloadTestSuite} + +/** + * Multi-operation write workloads: long op sequences, property churn, schema evolution chains, + * and high-level/low-level mixes. Stresses the commit-index==version invariant and time-travel + * across many versions. + */ +class WriteSequencesSuite extends WorkloadTestSuite("write_sequences") { + + test("insert_update_delete_chain") { + val w = createTableOp("tbl", schema = "id INT, status STRING, n INT", + properties = Map("delta.enableDeletionVectors" -> "true")) + insertOp(w, (1 to 3).map(i => Map("id" -> i, "status" -> "new", "n" -> 0))) + updateOp(w, predicate = "id <= 2", set = Map("status" -> "'active'", "n" -> "n + 1")) + deleteOp(w, predicate = "id = 3") + insertOp(w, Seq(Map("id" -> 4, "status" -> "new", "n" -> 5))) + updateOp(w, predicate = "status = 'new'", set = Map("n" -> "99")) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_v1") + readSpec(t, version = 2, name = "read_after_update") + readSpec(t, version = 3, name = "read_after_delete") + readSpec(t, predicate = "status = 'active'", name = "read_active") + readSpec(t, predicate = "n = 99", name = "read_n99") + snapshotSpec(t) + } + + test("property_churn") { + val w = createTableOp("tbl", schema = "id INT, v INT") + insertOp(w, Seq(Map("id" -> 1, "v" -> 1))) + setPropertiesOp(w, Map("delta.enableChangeDataFeed" -> "true")) + insertOp(w, Seq(Map("id" -> 2, "v" -> 2))) + unsetPropertiesOp(w, Seq("delta.enableChangeDataFeed")) + setPropertiesOp(w, Map("delta.enableChangeDataFeed" -> "true", + "delta.enableDeletionVectors" -> "true")) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("schema_evolution_chain") { + // Column mapping enables rename/drop; chain add -> rename -> drop with data between each. + val w = createTableOp("tbl", schema = "id INT, a STRING", + properties = Map("delta.columnMapping.mode" -> "name")) + insertOp(w, Seq(Map("id" -> 1, "a" -> "x"))) + addColumnsOp(w, "b INT") + insertOp(w, Seq(Map("id" -> 2, "a" -> "y", "b" -> 10))) + renameColumnOp(w, "a", "label") + insertOp(w, Seq(Map("id" -> 3, "label" -> "z", "b" -> 20))) + dropColumnsOp(w, Seq("b")) + insertOp(w, Seq(Map("id" -> 4, "label" -> "w"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 1, name = "read_initial") + readSpec(t, version = 3, name = "read_after_add") + readSpec(t, predicate = "label = 'x'", name = "read_renamed") + snapshotSpec(t) + } + + + test("domain_metadata_add_then_remove") { + val w = createTableOp("tbl", schema = "id INT, name STRING", + properties = Map("delta.feature.domainMetadata" -> "supported")) + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + commitOp(w, addDomainMetadata = Some(Seq( + AddDomainMetadata("delta.test", """{"v":1}""")))) + commitOp(w, removeDomainMetadata = Some(Seq("delta.test"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("low_and_high_level_mix") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + val added = commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + insertOp(w, Seq(Map("id" -> 3, "name" -> "cara"))) + deleteOp(w, predicate = "id = 1") + commitOp(w, removeFiles = Some(Seq(added))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, version = 2, name = "read_after_lowlevel_add") + snapshotSpec(t) + } + + test("commit_txn_and_files") { + val w = createTableOp("tbl", schema = "id INT, name STRING") + insertOp(w, Seq(Map("id" -> 1, "name" -> "alice"))) + // A single low-level commit carrying both a SetTransaction and an AddFile. + commitOp(w, + txn = Some(AppTxn("streaming-app", 42L)), + addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob")))))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } +} diff --git a/workload-generator/src/test/scala/io/delta/workload/tables/WriteTypesSuite.scala b/workload-generator/src/test/scala/io/delta/workload/tables/WriteTypesSuite.scala new file mode 100644 index 00000000..f70a2662 --- /dev/null +++ b/workload-generator/src/test/scala/io/delta/workload/tables/WriteTypesSuite.scala @@ -0,0 +1,131 @@ +/* + * Copyright (2025) The Delta Lake Project Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.delta.workload.tables + +import io.delta.workload.WorkloadTestSuite + +/** + * Type-coverage write workloads: exercises the harness's row-value coercion and the schema round-trip across + * every supported scalar type, nulls, and string edge cases. Each test captures then the + * framework replays + re-validates, so a passing test proves the type survives capture->replay. + */ +class WriteTypesSuite extends WorkloadTestSuite("write_types") { + + test("scalar_types") { + val w = createTableOp("tbl", + schema = "id INT, c_long BIGINT, c_short SMALLINT, c_byte TINYINT, " + + "c_float FLOAT, c_double DOUBLE, c_bool BOOLEAN, c_str STRING") + insertOp(w, Seq( + Map("id" -> 1, "c_long" -> 10000000000L, "c_short" -> 100, "c_byte" -> 5, + "c_float" -> 1.5f, "c_double" -> 2.5, "c_bool" -> true, "c_str" -> "a"), + Map("id" -> 2, "c_long" -> 20000000000L, "c_short" -> 200, "c_byte" -> 7, + "c_float" -> 3.5f, "c_double" -> 9.5, "c_bool" -> false, "c_str" -> "b"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "c_bool = true", name = "read_true") + readSpec(t, predicate = "c_double > 5.0", name = "read_big_double") + readSpec(t, predicate = "c_long > 15000000000", name = "read_big_long") + snapshotSpec(t) + } + + test("date_and_timestamp") { + val w = createTableOp("tbl", schema = "id INT, d DATE, ts TIMESTAMP, ntz TIMESTAMP_NTZ") + insertOp(w, Seq( + Map("id" -> 1, "d" -> "2021-01-15", "ts" -> "2021-01-15 10:30:00", + "ntz" -> "2021-06-01 08:00:00"), + Map("id" -> 2, "d" -> "2022-07-04", "ts" -> "2022-07-04 23:59:59", + "ntz" -> "2022-12-31 00:00:00"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "d > DATE '2021-06-01'", name = "read_after_date") + readSpec(t, predicate = "ts < TIMESTAMP '2022-01-01 00:00:00'", name = "read_before_ts") + snapshotSpec(t) + } + + test("decimal_type") { + val w = createTableOp("tbl", schema = "id INT, amount DECIMAL(10,2)") + insertOp(w, Seq( + Map("id" -> 1, "amount" -> "123.45"), + Map("id" -> 2, "amount" -> "99.99"), + Map("id" -> 3, "amount" -> "1000.00"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "amount > 100.00", name = "read_over_100") + snapshotSpec(t) + } + + test("binary_type") { + val w = createTableOp("tbl", schema = "id INT, data BINARY") + insertOp(w, Seq( + Map("id" -> 1, "data" -> "aGVsbG8="), // "hello" + Map("id" -> 2, "data" -> "d29ybGQ="))) // "world" + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + snapshotSpec(t) + } + + test("nullable_columns") { + val w = createTableOp("tbl", schema = "id INT, name STRING, score INT, d DATE") + insertOp(w, Seq( + Map("id" -> 1, "name" -> "alice", "score" -> 90, "d" -> "2021-01-01"), + Map("id" -> 2, "name" -> null, "score" -> null, "d" -> null), + Map("id" -> 3, "name" -> "bob", "score" -> 75, "d" -> null))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "name IS NULL", name = "read_null_name") + readSpec(t, predicate = "name IS NOT NULL", name = "read_named") + readSpec(t, predicate = "d IS NULL", name = "read_null_date") + snapshotSpec(t) + } + + test("string_edge_cases") { + val w = createTableOp("tbl", schema = "id INT, s STRING") + insertOp(w, Seq( + Map("id" -> 1, "s" -> "it's a \"quoted\" value"), + Map("id" -> 2, "s" -> "unicode: café ☕ — ünïcödé"), + Map("id" -> 3, "s" -> ""), + Map("id" -> 4, "s" -> "line1\nline2\ttab"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "s = ''", name = "read_empty") + readSpec(t, predicate = "id = 1", name = "read_apostrophe") + snapshotSpec(t) + } + + test("nested_type_data_is_unsupported") { + // Nested column TYPES round-trip in the schema, but nested DATA via the rows API is not + // supported — the harness's row-value coercion fails loud rather than silently writing a wrong value. + val w = createTableOp("tbl", schema = "id INT, s STRUCT") + val ex = intercept[IllegalArgumentException] { + insertOp(w, Seq(Map("id" -> 1, "s" -> Map("a" -> 1)))) + } + assert(ex.getMessage.toLowerCase.contains("unsupported"), s"message: ${ex.getMessage}") + } + + test("partitioned_by_typed_column") { + val w = createTableOp("tbl", + schema = "id INT, d DATE, amount DECIMAL(8,2)", partitionColumns = Seq("d")) + insertOp(w, Seq( + Map("id" -> 1, "d" -> "2021-01-01", "amount" -> "10.00"), + Map("id" -> 2, "d" -> "2021-01-01", "amount" -> "20.00"), + Map("id" -> 3, "d" -> "2021-02-01", "amount" -> "30.00"))) + val t = registerWriteSpec(w) + readSpec(t, name = "read_all") + readSpec(t, predicate = "d = DATE '2021-01-01'", name = "read_jan") + snapshotSpec(t) + } +}