Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions workload-generator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,12 @@ See the [Spec Format Reference](docs/spec-reference.md) for the complete JSON sc

The write DSL (`createTableOp`, `insertOp`, `commitOp`, …) is exposed by `WorkloadOps`; see the [Authoring Guide](docs/authoring-guide.md) for how to author write workloads. There is no per-spec `writeSpec` pointer: whether the `read`/`snapshot` specs in a directory validate against the captured `delta/` table or against a replay of the sibling write spec is decided per directory — if a `*_write.json` is present, they are validated *portably* against the replay (replay the write spec into a fresh table, then compare).

`create_table` commits include an optional `createSql` template for Spark/Delta SQL consumers.
It uses the generator's creation SQL with a destination-table placeholder; structured schema,
partition columns, and properties remain available to non-SQL consumers. See the
[creation SQL contract](docs/spec-reference.md#high-level-operations) for target binding and
the fallback when the field is absent.

## Workload Suites

Workload suites are in `src/test/scala/io/delta/workload/tables/`. Each suite extends `WorkloadTestSuite` and covers a specific Delta feature area (reads, deletion vectors, column mapping, time travel, checkpoints, data skipping, protocol versions, merge, schema evolution, type widening, variant, row tracking, in-commit timestamps, and more).
Expand Down
14 changes: 10 additions & 4 deletions workload-generator/docs/authoring-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,13 @@ Inside a test body, these methods are available directly (via `WorkloadOps` trai
| SQL | `sql(statement)` | — |
| Table handles | `registerTable(name)`, `registerTableFromPath(path)` | → `TableHandle` |
| Read specs | `readSpec(t, ...)`, `snapshotSpec(t, ...)` | `TableHandle` → `SpecRef` |
| Write specs | `createTableOp`, `replaceTableOp`, `insertOp`, `deleteOp`, `updateOp`, `addColumnsOp`, `renameColumnOp`, `dropColumnsOp`, `setPropertiesOp`, `unsetPropertiesOp`, `commitOp` (low-level) | → `WriteHandle`; `registerWriteSpec(w)` → `TableHandle` |
| Write specs | `createTableOp`, `replaceTableOp`, `insertOp`, `deleteOp`, `updateOp`, `addColumnsOp`, `renameColumnOp`, `dropColumnsOp`, `setPropertiesOp`, `unsetPropertiesOp`, `commitOp` (low-level) | → `WriteHandle`; `endWrite(w)` → `TableHandle` |
| Checkpointing | `forceCheckpoint(tableName)` — triggers a checkpoint via DeltaLog | — |
| Mutations | `mutateTable(t) { dir => ... }`, `modifyCommitActions(t, version) { ... }` | `TableHandle` |

### Authoring a write workload

A write workload builds a table through DSL ops instead of raw `sql(...)`, producing a portable `write` spec (`specs/<name>_write.json`) plus any `read`/`snapshot` specs derived from the built table. Start with `createTableOp` (or `replaceTableOp`), apply ops, then `registerWriteSpec(w)` to get a `TableHandle` you can attach reads/snapshots to:
A write workload builds a table through DSL ops instead of raw `sql(...)`, producing a portable `write` spec (`specs/<name>_write.json`) plus any `read`/`snapshot` specs derived from the built table. Start with `createTableOp`, apply ops, then `endWrite(w)` to get a `TableHandle` you can attach reads/snapshots to:

```scala
import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
Expand All @@ -108,15 +108,21 @@ val w = createTableOp("tbl", schema = schema)
insertOp(w, Seq(Map("id" -> 1, "name" -> "alice")))
// Low-level commit: logical rows are written through the engine (stats/partitionValues derived).
commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob"))))))
val t = registerWriteSpec(w)
readSpec(t, name = "read_all")
val t = endWrite(w)
readSpec(t, name = Some("read_all"))
snapshotSpec(t)
```

The write-op schema inputs (`createTableOp`, `replaceTableOp`, `addColumnsOp`, and `commitOp`'s
optional `schema`) are Spark `StructType`s, not DDL strings. The generator stores them as Delta
schema JSON in the spec.

`createTableOp` also exports its Spark/Delta creation SQL as `createSql`, with `{{table}}`
in place of the reference table's name. Authors do not need to supply SQL separately.
The schema, partition columns, and properties remain in the commit for consumers that
use their own table-creation API. Calling `sql("CREATE TABLE ...")` alone does not export
a creation template. See the [creation SQL contract](spec-reference.md#high-level-operations).

`commitOp` returns the commit's ordinal (== table version); a later `removeFiles = Some(Seq(ordinal))` tombstones the files that commit added. See `WriteBasicSuite`/`WriteCommitSuite`/`WriteSequencesSuite` for worked examples and the [Write Spec reference](spec-reference.md#write-spec) for the on-disk format.

---
Expand Down
21 changes: 21 additions & 0 deletions workload-generator/docs/harness-implementation-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,27 @@ Run the operation and assert it fails. Matching the exact error code is ideal bu

A `write` spec (`specs/<name>_write.json`) is a portable recipe for *building* a table: an ordered list of `commits` your harness replays into a fresh table. High-level commits (create/replace/insert/delete/update/schema-evolution/properties) are replayed through your engine's normal write APIs; low-level `commit`s carry raw actions (the data files hold logical rows your engine writes through its own write path, so physical names/stats are engine-derived, and removes reference a prior add by commit ordinal == table version). See the [Write Spec reference](spec-reference.md#write-spec) for the field-level contract.

For `create_table`, a Spark/Delta SQL consumer can execute the optional `createSql` template
instead of converting the structured schema to SQL. Bind only the target placeholder at the
start of the statement; leave placeholder text inside column definitions and literals intact.
For example, given a template and an already-quoted destination table reference:

```scala
val prefix = "CREATE TABLE {{table}}"
require(template.startsWith(prefix + " ("))
val statement = "CREATE TABLE " + targetTableRef + template.substring(prefix.length)
executeSql(statement)
```

Execute this statement through the harness's SQL setup engine. A separate row-ingestion API
still receives rows. If `createSql` is absent, construct the statement from `schema`,
`partitionColumns`, and `properties`; consumers without SQL support use those structured
fields directly. The template dialect and binding rules are specified in the
[creation SQL contract](spec-reference.md#high-level-operations).

Readback filters and projections use the existing read-spec `predicate` and `columns`
fields. `createSql` does not add SQL templates for reads or other write operations.

The write spec has no expected-data artifact of its own. Its own check is **basic**: the replay must succeed and produce the expected number of versions (`finalVersion == commits.size - 1`). The final-state rows are validated by an auto-generated baseline read spec named `latest` (`specs/<name>_latest.json` with rows under `expected/<name>_latest/expected_data/`), and per-version protocol/metadata by the snapshot spec(s).

Because a write spec is replayed (not byte-compared), comparison is **portable**: rows-only data checks, capability-based protocol comparison (your table may use explicit table features where Spark used legacy versions), and column-mapping-normalized schema. A write spec's presence makes its whole directory *write-derived*: replay it into a fresh table once, then validate every read/snapshot spec in that directory (including the baseline `latest` read) against the replay (portably), rather than against a captured `delta/` directory.
Expand Down
17 changes: 15 additions & 2 deletions workload-generator/docs/spec-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,14 +375,26 @@ These map to SQL-like operations. The writer translates them to appropriate Delt

| Operation | Fields | Description |
|-----------|--------|-------------|
| `create_table` | `schema`, `partitionColumns?`, `properties?` | Create a new table with the given schema (`schema` is a Delta-JSON struct) |
| `create_table` | `schema`, `partitionColumns?`, `properties?`, `createSql?` | Create a new table with the given schema (`schema` is a Delta-JSON struct); `createSql` optionally supplies the Spark SQL creation template |
| `replace_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Replace the table's schema/partitioning/properties (and all data). With `dataFiles` it is a replace-as-select: a single commit that also writes the bundled data. |
| `insert` | `dataFiles?` | Append the rows in the bundled Parquet data files |
| `update` | `predicate`, `set` | Update rows matching `predicate`; `set` maps column → SQL expression |
| `delete` | `predicate` | Delete rows matching `predicate` (SQL WHERE clause) |
| `evolve_schema` | `addColumns?`, `renameColumns?`, `dropColumns?` | Modify table schema. `addColumns` entries are `{name, type, nullable}`; `renameColumns` maps old → new |
| `update_properties` | `set?`, `remove?` | Modify table properties (`set` map, `remove` names) |

`createTableOp` exports `createSql` using the same SQL template it executes to create the
reference table. The template starts with `CREATE TABLE {{table}} (`. SQL consumers replace
only that target placeholder with their own quoted table identifier or Delta path reference,
then execute the statement. Placeholder text inside column definitions, comments, or property
values must remain unchanged. The template contains the column definitions, partition clause,
and table properties; it does not contain the reference table's name or storage location.

The structured `schema`, `partitionColumns`, and `properties` remain available to consumers
that do not execute SQL. Older corpora can omit `createSql`; SQL consumers then construct the
creation statement from the structured fields. The exported template uses Spark/Delta SQL
and must agree with the structured definition. Other operations do not export SQL templates.

The data for `insert` and `replace_table` is bundled as Parquet under `data/commit_N/` (the
generator's authoring API accepts in-memory rows, but the spec always stores Parquet). A
consumer replays by reading those files — `insert` appends them; `replace_table` performs a
Expand Down Expand Up @@ -464,7 +476,8 @@ The write spec itself only asserts that the replay succeeds and produces the exp
{ "name": "status", "type": "string", "nullable": true, "metadata": {} }
]
},
"properties": { "delta.enableDeletionVectors": "true" }
"properties": { "delta.enableDeletionVectors": "true" },
"createSql": "CREATE TABLE {{table}} (id INT NOT NULL, status STRING) USING delta TBLPROPERTIES ('delta.enableDeletionVectors' = 'true')"
},
{ "operation": "insert", "dataFiles": ["data/commit_1/part-0000-abc.parquet"] },
{ "operation": "delete", "predicate": "id > 100" },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,15 @@ trait WorkloadOps {
properties: Map[String, String] = Map.empty,
partitionColumns: Seq[String] = Seq.empty): WriteHandle = {
val ctx = WorkloadContext.current
ctx.sql(TableSql.createTable(tableName, schema, partitionColumns, properties))
val createSql = TableSql.createTable(
TableSql.tablePlaceholder, schema, partitionColumns, properties)
ctx.sql(TableSql.bindCreateTable(createSql, tableName))
val t = ctx.registerTable(tableName)
ctx.getWriteBuilder(t).record(CreateTableCommit(
schema = schema,
partitionColumns = ctx.toOption(partitionColumns),
properties = ctx.toOption(properties)))
properties = ctx.toOption(properties),
createSql = Some(createSql)))
new WriteHandle(t)
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,12 +101,14 @@ sealed trait WriteCommit {

// ---- High-level operations (replayed from parameters) ----

@JsonPropertyOrder(Array("schema", "partitionColumns", "properties"))
/** createSql optionally supplies Spark SQL for the same definition, with a {{table}} target. */
@JsonPropertyOrder(Array("schema", "partitionColumns", "properties", "createSql"))
@JsonInclude(JsonInclude.Include.NON_ABSENT)
case class CreateTableCommit(
schema: StructType,
partitionColumns: Option[Seq[String]] = None,
properties: Option[Map[String, String]] = None) extends WriteCommit {
properties: Option[Map[String, String]] = None,
createSql: Option[String] = None) extends WriteCommit {
@JsonIgnore val operation = "create_table"
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,33 @@ import org.apache.spark.sql.types.StructType
*/
private[workload] object TableSql {

val tablePlaceholder: String = "{{table}}"

/** `CREATE [OR REPLACE] TABLE ref (cols) USING delta [PARTITIONED BY …] [TBLPROPERTIES …]`. */
def createTable(
ref: String, schema: StructType, partitionColumns: Seq[String],
properties: Map[String, String], orReplace: Boolean = false): String =
s"CREATE ${if (orReplace) "OR REPLACE " else ""}TABLE $ref (${schema.toDDL}) USING delta" +
s"${createTablePrefix(ref, orReplace)} (${schema.toDDL}) USING delta" +
partitionedBy(partitionColumns) + tblProperties(properties)

/** Bind the CREATE TABLE target without replacing placeholder text inside SQL literals. */
def bindCreateTable(sql: String, ref: String): String = {
val prefix = createTablePrefix(tablePlaceholder)
require(sql.startsWith(prefix + " ("),
s"createSql must start with '$prefix ('")
s"${createTablePrefix(ref)}${sql.substring(prefix.length)}"
}
Comment thread
rliao147 marked this conversation as resolved.
Resolved

/** `CREATE OR REPLACE TABLE ref USING delta [PARTITIONED BY …] [TBLPROPERTIES …] AS <select>` (RTAS). */
def replaceTableAsSelect(
ref: String, select: String, partitionColumns: Seq[String],
properties: Map[String, String]): String =
s"CREATE OR REPLACE TABLE $ref USING delta" +
partitionedBy(partitionColumns) + tblProperties(properties) + s" AS $select"

private def createTablePrefix(ref: String, orReplace: Boolean = false): String =
s"CREATE ${if (orReplace) "OR REPLACE " else ""}TABLE $ref"

private def partitionedBy(columns: Seq[String]): String =
if (columns.nonEmpty) s" PARTITIONED BY (${columns.mkString(", ")})" else ""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,12 @@ object WriteReplay {
spark: SparkSession, commit: WriteCommit, idx: Int,
tableRef: String, tablePath: String, testDir: Path): Unit = commit match {
case c: CreateTableCommit =>
spark.sql(TableSql.createTable(tableRef, c.schema,
c.partitionColumns.getOrElse(Nil), c.properties.getOrElse(Map.empty)))
val sql = c.createSql match {
case Some(template) => TableSql.bindCreateTable(template, tableRef)
case None => TableSql.createTable(tableRef, c.schema,
c.partitionColumns.getOrElse(Nil), c.properties.getOrElse(Map.empty))
}
spark.sql(sql)

case c: ReplaceTableCommit =>
// Always replace via AS SELECT: `CREATE OR REPLACE TABLE delta.`path` (cols)` is treated as
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,19 @@ package io.delta.workload

import java.nio.file.{Files, Path}

import scala.collection.immutable.ListMap

import com.fasterxml.jackson.databind.node.ObjectNode
import org.apache.commons.io.FileUtils
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
import org.scalatest.BeforeAndAfterAll
import org.scalatest.funsuite.AnyFunSuite

import io.delta.workload.json.JsonUtil
import io.delta.workload.model._
import io.delta.workload.validate.WorkloadValidator
import io.delta.workload.write.WriteReplay

/**
* Verifies [[WorkloadValidator]] can re-validate a workload tree produced by
Expand Down Expand Up @@ -120,6 +125,61 @@ class WorkloadValidatorSuite extends AnyFunSuite with BeforeAndAfterAll with Wor
s"expected >= 3 specs (write + read + snapshot), got ${result.passed}")
}

test("write replay executes the supplied creation SQL") {
val testDir = Files.createDirectory(outputDir.resolve("sql_replay"))
val specPath = testDir.resolve("create_write.json")
val schema = new StructType().add("id", IntegerType)
JsonUtil.writeSpec(specPath, WriteSpec(Seq(CreateTableCommit(schema,
createSql = Some("CREATE TABLE {{table}} (id INT) USING delta " +
"COMMENT 'from exported SQL'")))))
val replayPath = testDir.resolve("replayed")
WriteReplay.replayInto(spark, testDir, specPath, replayPath)
val description = spark.sql(s"DESCRIBE DETAIL delta.`${replayPath.toAbsolutePath}`")
.select("description").head().getString(0)
assert(description == "from exported SQL")
}

Seq(true, false).foreach { includeSql =>
test(s"validates partitioned nested writes with createSql present=$includeSql") {
val name = s"val_create_sql_$includeSql"
val tableName = s"create_sql_$includeSql"
spark.sql(s"DROP TABLE IF EXISTS $tableName")
generate(name) {
val schema = new StructType()
.add("id", IntegerType, nullable = false)
.add("region", StringType)
.add("details", new StructType().add("labels", ArrayType(StringType)))
val w = createTableOp(tableName, schema,
partitionColumns = Seq("region"),
properties = ListMap("delta.appendOnly" -> "true", "note" -> "{{table}}"))
insertOp(w, Seq(Map(
"id" -> 1, "region" -> "east",
"details" -> Map("labels" -> Seq("a", "b")))))
val t = endWrite(w)
readSpec(t)
snapshotSpec(t)
}

val testDir = outputDir.resolve(name)
val specPath = testDir.resolve("specs").resolve(s"${name}_write.json")
val spec = JsonUtil.mapper.readTree(Files.readAllBytes(specPath))
val create = spec.path("commits").get(0).asInstanceOf[ObjectNode]
val createSql = create.path("createSql").asText()
assert(createSql == "CREATE TABLE {{table}} (id INT NOT NULL,region STRING," +
"details STRUCT<labels: ARRAY<STRING>>) USING delta PARTITIONED BY (region) " +
"TBLPROPERTIES ('delta.appendOnly' = 'true', 'note' = '{{table}}')")
if (!includeSql) {
create.remove("createSql")
Files.write(specPath, JsonUtil.mapper.writeValueAsBytes(spec))
}

val result = WorkloadValidator.validateTestDir(spark, testDir)
assert(result.success,
s"createSql present=$includeSql; errors:\n ${result.errors.mkString("\n ")}")
assert(result.passed >= 3)
}
}

test("validator reports failures when the table diverges from the spec") {
spark.sql("DROP TABLE IF EXISTS tbl2")
generate("val_divergence") {
Expand Down
Loading
Loading