Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
284838d
feat: add write workload generator and spec
rliao147 Jun 15, 2026
f527c17
refactor: route write-spec replay through the DeltaHarness SPI
rliao147 Jun 15, 2026
93c83eb
refactor: unify write into a single spec + validator, replay low-leve…
OussamaSaoudi-db Jun 17, 2026
11915e0
refactor: stop re-validating Spark's own output; enforce invariants a…
OussamaSaoudi-db Jun 17, 2026
b28d39e
refactor: route commit-log mutation through the typed log/ module
OussamaSaoudi-db Jun 17, 2026
ab29328
refactor: split the WorkloadGenerator god-file by concern
OussamaSaoudi-db Jun 17, 2026
095f43d
perf: validate test dirs concurrently in validateAll
OussamaSaoudi-db Jun 17, 2026
fe1d92c
style: drop em-dashes from comments per house style
OussamaSaoudi-db Jun 17, 2026
b8af516
fix: validate row count when expected_data exceeds the materializatio…
OussamaSaoudi-db Jun 17, 2026
8fa6e67
refactor: address review follow-ups (addPathsAt via log/, doc fidelit…
OussamaSaoudi-db Jun 17, 2026
a56f512
fix: replay evolve_schema via ALTER TABLE so the engine enforces colu…
OussamaSaoudi-db Jun 17, 2026
e0731a3
docs: fix WriteCommit field reference to match the sealed ADT
OussamaSaoudi-db Jun 17, 2026
7cade76
feat: insertOp/replaceTableOp accept IterableOnce; generate formulaic…
OussamaSaoudi-db Jun 17, 2026
9f89635
test: collapse verbose row literals into iterators (minimize LOC)
OussamaSaoudi-db Jun 17, 2026
6639580
fix: restore write-derived snapshot validation
OussamaSaoudi-db Jun 17, 2026
0006583
refactor: move row->Parquet materialization onto the DeltaHarness SPI
OussamaSaoudi-db Jun 17, 2026
3ea19ec
style: revert TableInfoWriter reindent (whitespace-only churn)
OussamaSaoudi-db Jun 17, 2026
227e4ee
refactor: collapse WriteSpecBuilder record* methods into one generic …
OussamaSaoudi-db Jun 17, 2026
4ff5b6c
refactor: keep the workload engine/DSL in one file (drop the god-file…
OussamaSaoudi-db Jun 17, 2026
2380f22
docs: trim the write-spec reference
OussamaSaoudi-db Jun 17, 2026
6732750
refactor: carry write-op rows with the commit instead of side maps
OussamaSaoudi-db Jun 17, 2026
eff054f
refactor: fold addDataParquet into CommitRequest, collapse to one com…
OussamaSaoudi-db Jun 18, 2026
5801758
refactor: move writeRowsToTemp off the SPI, drop schemaAt includePart…
OussamaSaoudi-db Jun 19, 2026
6a827a5
chore: reword comments flagged by the typos linter
rliao147 Jun 22, 2026
40c1bad
fix: use Iterable instead of IterableOnce for scala 2.12 build
rliao147 Jun 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions workload-generator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Write a script that creates Delta tables with normal SQL, declare what specs to

| Document | Description |
|----------|-------------|
| **[Spec Format Reference](docs/spec-reference.md)** | JSON schema for the `read` and `snapshot` spec types, with examples |
| **[Spec Format Reference](docs/spec-reference.md)** | JSON schema for the `read`, `snapshot`, and `write` spec types, with examples |
| **[Harness Implementation Guide](docs/harness-implementation-guide.md)** | Step-by-step guide to build a test harness that runs workloads against your engine, with a Rust example |
| **[Authoring Guide](docs/authoring-guide.md)** | How to write new workload suites, patterns, recipes, and debugging tips |

Expand Down Expand Up @@ -81,8 +81,10 @@ WORKLOAD_FORCE=true WORKLOAD_OUTPUT_DIR=/tmp/workloads sbt "testOnly *ReadsSuite
| Component | Purpose |
|-----------|---------|
| `WorkloadTestSuite` | ScalaTest base class with workload generation integration |
| `WorkloadOps` | DSL trait: `sql()`, `registerTable()`, `readSpec()`, `snapshotSpec()` |
| `WorkloadGenerator` | Orchestrates table copy, spec capture, and validation |
| `WorkloadOps` | DSL trait (pass-through over `WorkloadContext`): `sql()`, `registerTable()`, `readSpec()`, write ops |
| `WorkloadContext` | DSL state: tracks created tables, declared specs, and write builders |
| `WorkloadGenerator` | Orchestrates table copy, spec capture, and self-validation |
| `WorkloadValidator` | Replays/validates a generated workload tree; presubmit acceptance entry point |
| `ReadCapture` | Captures read specs with expected row data |
| `SnapshotCapture` | Captures snapshot specs with protocol/metadata |
| `TableInfoWriter` | Writes table metadata (schema, protocol, stats) |
Expand Down Expand Up @@ -157,6 +159,9 @@ See the [Spec Format Reference](docs/spec-reference.md) for the complete JSON sc
|------|--------------|---------|
| **Read** | Data reads with time travel, predicates, column projection, data skipping | [Reference](docs/spec-reference.md#read-spec) |
| **Snapshot** | Protocol and metadata reconstruction from log replay | [Reference](docs/spec-reference.md#snapshot-spec) |
| **Write** | Table construction via a portable sequence of commits (create/replace/insert/delete/update/schema-evolution/properties and low-level `commit`s), replayed into a fresh table | [Reference](docs/spec-reference.md#write-spec) |

The write DSL (`createTableOp`, `insertOp`, `commitOp`, …) is exposed by `WorkloadOps`; see the [Authoring Guide](docs/authoring-guide.md) for how to author write workloads. A `read`/`snapshot` spec captured against a write-built table carries a `writeSpec` pointer and is validated *portably* (replay the sibling write spec into a fresh table, then compare).

## Workload Suites

Expand Down
34 changes: 25 additions & 9 deletions workload-generator/docs/authoring-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,26 @@ Inside a test body, these methods are available directly (via `WorkloadOps` trai
| SQL | `sql(statement)` | — |
| Table handles | `registerTable(name)`, `registerTableFromPath(path)` | → `TableHandle` |
| Read specs | `readSpec(t, ...)`, `snapshotSpec(t, ...)` | `TableHandle` → `SpecRef` |
| Write specs | `createTableOp`, `replaceTableOp`, `insertOp`, `deleteOp`, `updateOp`, `addColumnsOp`, `renameColumnOp`, `dropColumnsOp`, `setPropertiesOp`, `unsetPropertiesOp`, `commitOp` (low-level) | → `WriteHandle`; `registerWriteSpec(w)` → `TableHandle` |
| Checkpointing | `forceCheckpoint(tableName)` — triggers a checkpoint via DeltaLog | — |
| Mutations | `mutateTable(t) { dir => ... }`, `modifyCommitActions(t, version) { ... }` | `TableHandle` |

### Authoring a write workload

A write workload builds a table through DSL ops instead of raw `sql(...)`, producing a portable `write` spec (`specs/<name>_write.json`) plus any `read`/`snapshot` specs derived from the built table. Start with `createTableOp` (or `replaceTableOp`), apply ops, then `registerWriteSpec(w)` to get a `TableHandle` you can attach reads/snapshots to:

```scala
val w = createTableOp("tbl", schema = "id INT, name STRING")
insertOp(w, Seq(Map("id" -> 1, "name" -> "alice")))
// Low-level commit: logical rows are written through the engine (stats/partitionValues derived).
commitOp(w, addFiles = Some(Seq(AddFileInput(rows = Seq(Map("id" -> 2, "name" -> "bob"))))))
val t = registerWriteSpec(w)
readSpec(t, name = "read_all")
snapshotSpec(t)
```

`commitOp` returns the commit's ordinal (== table version); a later `removeFiles = Some(Seq(ordinal))` tombstones the files that commit added. See `WriteBasicSuite`/`WriteCommitSuite`/`WriteSequencesSuite` for worked examples and the [Write Spec reference](spec-reference.md#write-spec) for the on-disk format.

---

## Creating Tables
Expand Down Expand Up @@ -311,23 +328,22 @@ test("missing_file", "Test missing data file") {

### modifyCommitActions — Action-Level Manipulation

The modifier receives the full list of `(actionType, innerNode)` pairs for a commit and returns the (possibly reordered, filtered, or modified) list to write back.
The modifier receives the commit's typed `Seq[Action]` (subtypes `AddFile`, `RemoveFile`, `Metadata`, `Protocol`, `Txn`, `DomainMetadata`, …; unrecognized or malformed lines arrive as `RawAction` and pass through unchanged) and returns the reordered, filtered, or modified list to write back. Use `.copy` to edit fields. For byte-exact corruption, use the raw `mutateTable` path below instead.

```scala
import io.delta.workload.log.{AddFile, Metadata}

// Modify add actions
modifyCommitActions(t, version = 1) { actions =>
actions.map { case ("add", node) =>
node.put("stats", """{"numRecords":999}"""); ("add", node)
case other => other
}
modifyCommitActions(t, version = 1) {
_.map { case a: AddFile => a.copy(stats = Some("""{"numRecords":999}""")); case other => other }
}

// Drop all add actions
modifyCommitActions(t, version = 1) { _.filter(_._1 != "add") }
modifyCommitActions(t, version = 1) { _.filterNot(_.isInstanceOf[AddFile]) }

// Reorder — put metaData last
// Reorder: put metaData last
modifyCommitActions(t, version = 0) { actions =>
val (meta, rest) = actions.partition(_._1 == "metaData")
val (meta, rest) = actions.partition(_.isInstanceOf[Metadata])
rest ++ meta
}
```
Expand Down
6 changes: 6 additions & 0 deletions workload-generator/docs/harness-implementation-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,12 @@ Any spec type can have `"expectedError"` instead of `"expected"`:

Run the operation and assert it fails. Matching the exact error code is ideal but optional — just asserting failure is a valid starting point.

### Write Specs

A `write` spec (`specs/<name>_write.json`) is a portable recipe for *building* a table: an ordered list of `commits` your harness replays into a fresh table, then compares against `expected/<name>_write/`. High-level commits (create/replace/insert/delete/update/schema-evolution/properties) are replayed through your engine's normal write APIs; low-level `commit`s carry raw actions (the data files hold logical rows your engine writes through its own write path, so physical names/stats are engine-derived, and removes reference a prior add by commit ordinal == table version). See the [Write Spec reference](spec-reference.md#write-spec) for the field-level contract.

Because a write spec is replayed (not byte-compared), comparison is **portable**: rows-only data checks, capability-based protocol comparison (your table may use explicit table features where Spark used legacy versions), and column-mapping-normalized schema. A `read`/`snapshot` spec carrying a `writeSpec` pointer is *write-derived* — reconstruct the table by replaying that sibling write spec first, then validate the read/snapshot against it (portably), rather than against a captured `delta/` directory.

---

## Step 4: Incremental Adoption
Expand Down
206 changes: 206 additions & 0 deletions workload-generator/docs/spec-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Exactly one of `expected` or `expectedError` is present. The other is omitted (n
- [Common Types](#common-types)
- [Read Spec](#read-spec)
- [Snapshot Spec](#snapshot-spec)
- [Write Spec](#write-spec)
- [table_info.json](#table_infojson)
- [Expected Data Layout](#expected-data-layout)

Expand Down Expand Up @@ -80,6 +81,7 @@ Tests reading data from a Delta table with optional time travel, predicate pushd
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `string` | yes | Always `"read"` |
| `writeSpec` | `string` | no | If present, this read is *write-derived*: the named sibling write spec (`specs/<writeSpec>`) must be replayed into a fresh table first, then this read validated against it. Absent ⇒ read-only (validate against the captured table). |
| `version` | `long` | no | Time travel to this version |
| `timestamp` | `string` | no | Time travel to this timestamp (format: `yyyy-MM-dd HH:mm:ss.SSS`) |
| `predicate` | `string` | no | SQL WHERE clause to apply (e.g., `"id > 5"`) |
Expand Down Expand Up @@ -243,6 +245,7 @@ Tests constructing a table snapshot at a given version, validating the protocol
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `string` | yes | Always `"snapshot"` |
| `writeSpec` | `string` | no | If present, write-derived: replay the named sibling write spec first, then validate this snapshot against it (portable comparison). Absent ⇒ validate against the captured table. |
| `version` | `long` | no | Snapshot at this version (latest if omitted and no timestamp) |
| `timestamp` | `string` | no | Snapshot at this timestamp |
| `expected` | `SnapshotExpected` | no | Present on success |
Expand Down Expand Up @@ -347,6 +350,209 @@ These are the raw Delta protocol and metadata JSON structures — not simplified

---

## Write Spec

**Type:** `"write"`

Tests Delta *writer* implementations: a portable, engine-agnostic recipe of `commits` to replay into a fresh table, after which the result is compared against `expected/<name>_write/`.

It is **just another spec** — it lives at `specs/<name>_write.json` and dispatches by `type`. The read/snapshot specs of the same workload carry a `writeSpec` pointer back to it, marking them *write-derived* (replayed from the write spec, then validated against it).

### Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `string` | yes | Always `"write"` |
| `commits` | `WriteCommit[]` | yes | Ordered list of commits to replay |

### WriteCommit

Each commit represents a single Delta transaction. There are two categories: high-level operations (SQL semantics) and the low-level `commit` (raw Delta actions).

#### High-Level Operations

These map to SQL-like operations. The writer translates them to appropriate Delta actions.

| Operation | Fields | Description |
|-----------|--------|-------------|
| `create_table` | `schema`, `partitionColumns?`, `properties?` | Create a new table with the given schema (`schema` is a Delta-JSON struct) |
| `replace_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Replace the table's schema/partitioning/properties (and all data). With `dataFiles` it is a replace-as-select: a single commit that also writes the bundled data. |
| `insert` | `dataFiles?` | Append the rows in the bundled Parquet data files |
| `update` | `predicate`, `set` | Update rows matching `predicate`; `set` maps column → SQL expression |
| `delete` | `predicate` | Delete rows matching `predicate` (SQL WHERE clause) |
| `evolve_schema` | `addColumns?`, `renameColumns?`, `dropColumns?` | Modify table schema. `addColumns` entries are `{name, type, nullable}`; `renameColumns` maps old → new |
| `update_properties` | `set?`, `remove?` | Modify table properties (`set` map, `remove` names) |

The data for `insert` and `replace_table` is bundled as Parquet under `data/commit_N/` (the
generator's authoring API accepts in-memory rows, but the spec always stores Parquet). A
consumer replays by reading those files — `insert` appends them; `replace_table` performs a
single replace-as-select from them.

`RENAME COLUMN` / `DROP COLUMN` in `evolve_schema` require the table to have column mapping
enabled (a Delta/Spark constraint).

#### Low-Level Operation

| Operation | Fields | Description |
|-----------|--------|-------------|
| `commit` | `schema?`, `tableProperties?`, `txn?`, `addFiles?`, `removeFiles?`, `addDomainMetadata?`, `removeDomainMetadata?` | Bundle raw Delta actions in one commit. Each `addFiles[]` entry's logical rows are bundled as Parquet (`dataFile`); a consumer writes them **through its own engine write path** (so column mapping, partitioning, and stats are handled — physical names/stats are not in the spec) and commits the resulting `AddFile`s alongside the `txn`/`DomainMetadata`/schema/property changes. `removeFiles[]` tombstone a file added by a prior low-level `commit`, referenced by that commit's ordinal (`addedAtCommit`); since the engine assigns paths per table, the consumer resolves the ordinal to its own table's path. Deletion vectors are out of scope. |

### Low-level value types

Value types referenced by the low-level `commit` operation. An `AddFileAction` carries a pointer to a Parquet of **logical** rows (full rows incl. partition columns); the consumer writes it through its own engine write path, so physical names, partitioning, and stats are derived per table, not stored.

| Type | Field | Description |
|------|-------|-------------|
| `AddFileAction` | `dataFile` | Relative path under `data/commit_N/` to the Parquet of logical rows |
| `AddFileAction` | `dataChange?` | Whether this is a data change (default true) |
| `RemoveFileAction` | `addedAtCommit` | Ordinal (== table version) of the prior low-level `commit` that added the file(s) to tombstone; consumer resolves it to its own table's path(s) |
| `RemoveFileAction` | `dataChange?` | Whether this is a data change (default true) |
| `AppTxn` | `appId`, `version` | Application id and its monotonic transaction version for idempotent writes (`txn` action) |
| `AddDomainMetadata` | `domain`, `configuration` | Domain name and its configuration payload |

### Expected Data

The final table state after replaying all commits is captured under `expected/<name>_write/`:

- `table_content/` — Parquet files with the expected rows
- `table_version_metadata.json` — the latest snapshot's `protocol` and `metadata`

### Comparison Semantics

The capture comes from one writer, so a consumer compares the replayed table against `expected` *portably*, not byte-for-byte:

- **rows** (`table_content/`) — multiset equality (order-independent).
- **protocol** — the replay must support at least the expected reader/writer versions and features (a stronger protocol is acceptable).
- **configuration** — only the keys the spec's own commits declared: `create_table`/`replace_table` `properties` + `update_properties.set` minus `.remove` (`replace_table` resets the set); each present and equal, removed keys absent. Engine-injected defaults are ignored.
- **schemaString** — equal with per-field column-mapping `physicalName`/`id` normalized out (minted per table); `partitionColumns` and `format` equal.

### Directory Structure

```
<test_name>/
├── delta/ # the captured table (for read-only validation)
├── specs/
│ ├── <test>_read*.json
│ ├── <test>_snapshot.json
│ └── <test>_write.json # the write spec — just another spec
├── data/ # Parquet referenced by write commits
│ ├── commit_1/
│ │ └── part-00000.parquet
│ └── ...
├── expected/
│ ├── <test>_write/ # write spec's final state
│ │ ├── table_content/
│ │ └── table_version_metadata.json
│ └── <read_spec_name>/ # expected_data for each read spec
│ └── expected_data/
└── table_info.json
```

### Examples

**High-level sequence (create, insert, delete, update):**

```json
{
"type": "write",
"commits": [
{
"operation": "create_table",
"schema": {
"type": "struct",
"fields": [
{ "name": "id", "type": "integer", "nullable": false, "metadata": {} },
{ "name": "status", "type": "string", "nullable": true, "metadata": {} }
]
},
"properties": { "delta.enableDeletionVectors": "true" }
},
{ "operation": "insert", "dataFiles": ["data/commit_1/part-0000-abc.parquet"] },
{ "operation": "delete", "predicate": "id > 100" },
{
"operation": "update",
"predicate": "status = 'pending'",
"set": { "status": "'active'" }
}
]
}
```

**Schema evolution (add column):**

```json
{
"type": "write",
"commits": [
{
"operation": "create_table",
"schema": {
"type": "struct",
"fields": [
{ "name": "id", "type": "integer", "nullable": false, "metadata": {} }
]
}
},
{ "operation": "insert", "dataFiles": ["data/commit_1/part-0000-abc.parquet"] },
{
"operation": "evolve_schema",
"addColumns": [ { "name": "email", "type": "string", "nullable": true } ]
},
{ "operation": "insert", "dataFiles": ["data/commit_3/part-0000-def.parquet"] }
]
}
```

**Replace-as-select (replace schema + data in one commit):**

```json
{
"type": "write",
"commits": [
{ "operation": "create_table", "schema": { } },
{ "operation": "insert", "dataFiles": ["data/commit_1/part-00000.parquet"] },
{
"operation": "replace_table",
"schema": {
"type": "struct",
"fields": [
{ "name": "id", "type": "integer", "nullable": true, "metadata": {} },
{ "name": "label", "type": "string", "nullable": true, "metadata": {} }
]
},
"dataFiles": ["data/commit_2/part-00000.parquet"]
}
]
}
```

**Low-level commit (addFiles/removeFiles + txn + domainMetadata):**

```json
{
"type": "write",
"commits": [
{
"operation": "create_table",
"schema": { },
"properties": { "delta.feature.domainMetadata": "supported" }
},
{ "operation": "commit", "addFiles": [ { "dataFile": "data/commit_1/add_0.parquet" } ] },
{
"operation": "commit",
"txn": { "appId": "streaming-job-1", "version": 42 },
"addFiles": [ { "dataFile": "data/commit_2/add_0.parquet" } ],
"removeFiles": [ { "addedAtCommit": 1 } ],
"addDomainMetadata": [
{ "domain": "myApp.config", "configuration": "{\"version\": 1}" }
]
}
]
}
```

---

## table_info.json

Written to the workload output root. Provides metadata about the generated table for discovery and filtering.
Expand Down
Loading
Loading