Skip to content

feat: add write workload generator and spec - #55

Closed
rliao147 wants to merge 25 commits into
mainfrom
stack/add-write-generator
Closed

rliao147 wants to merge 25 commits into
mainfrom
stack/add-write-generator

Conversation

@rliao147

@rliao147 rliao147 commented Jun 15, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


Summary

DAT workloads only test readers (read / snapshot specs). This PR adds a write
spec: write_spec.json contains commits that any Delta writer can replay through its own write API, then validate against existing read/snapshot specs.

  1. Includes (create_table, insert, update, delete, add-column schema evolution, set/unset properties, restore) and commit for raw add/remove-file, SetTransaction, and DomainMetadata actions.
  2. WriteSpecCapture records each op and copies the data files it produces
  3. WriteSpecValidator replays every commit in write_spec.json into a fresh table and verifies the final table against expected/latest.

Co-author credit to @OussamaSaoudi since most of the code was ported over from his local fork.

@rliao147
rliao147 requested a review from OussamaSaoudi June 15, 2026 23:05
@rliao147

Copy link
Copy Markdown
Collaborator Author

Recommended review order:

  1. docs/spec-reference.md
  2. WriteBasicSuite.scala
  3. WorkloadGenerator.scala: the authoring API the suite calls
  4. JsonUtil.scala: the spec model (WriteSpec/WriteCommit/AddFileAction/…).
  5. WriteSpecCapture.scala: records ops, copies data to data/commit_N/, writes expected/latest.
  6. LowLevelCommit.scala: the shared capture+replay core for low-level commit (raw actions).
  7. WriteSpecValidator.scala: replays every commit in write_spec.json into a fresh table and verifies the final table against expected/latest.
  8. WriteCommitSuite.scala: low-level commit cases (txn/domainMetadata/addFiles/removeFiles) with independent _delta_log assertions.

@rliao147
rliao147 requested a review from nicklan June 15, 2026 23:08
| `delete` | `predicate` | Delete rows matching predicate |
| `evolve_schema` | `addColumns?`, `renameColumns?`, `dropColumns?` | Modify table schema |
| `update_properties` | `set?`, `remove?` | Modify table properties |
| `restore` | `version` | Restore table to a previous version |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be worth removing restore for now since I don't think it's really in demand, and this reference validator asterisk is a little concerning.


| Operation | Fields | Description |
|-----------|--------|-------------|
| `create_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Create a new table with the given schema |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should definitely include Replace tale. That has been the trickiest, yuckiest part of table formats and will be super useful for us to get the semantics right.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be done as a followup, but given that we're doing it all in one go, I'd prefer we do it eagerly if we can.

|-----------|--------|-------------|
| `create_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Create a new table with the given schema |
| `insert` | `dataFiles?` | Append rows from data files |
| `update` | `predicate`, `set` | Update rows matching predicate |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `update` | `predicate`, `set` | Update rows matching predicate |
| `update` | `predicate`, `set` | For each row matching the predicate, set it to at value |

^ Something like thi smight be more precise

OussamaSaoudi pushed a commit that referenced this pull request Jun 17, 2026
… split)

Fold WorkloadContext and WorkloadOps back into WorkloadGenerator.scala so PR #55
is the write-generator feature, not a file reorganization. Pure code movement (no
behavior change); the split will land as its own follow-up PR. Cuts ~1k churn
lines from the PR.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
rliao147 and others added 22 commits June 22, 2026 23:04
This PR adds a write spec, which allows DAT workloads to test Delta
writers. Any Delta writer can validate their own write APIs through the
existing read/snapshot specs.
…l commits via the engine

Make "everything a spec": one WorkloadValidator validates read/snapshot/write,
dispatching on `type`. Read/snapshot specs carry an optional `writeSpec` pointer
marking them write-derived (reconstruct the table by replaying the sibling write
spec, then validate portably) vs read-only (validate the captured delta/ exactly).
Deletes the separate WriteSpecValidator.

Model the write spec as a sealed WriteCommit ADT (one *Commit case class per op),
adding replace_table (incl. replace-as-select) and removing restore. Low-level
`commit` adds carry logical rows materialized to Parquet and written through the
engine's OptimisticTransaction.writeFiles, so column mapping / partitioning / stats
are engine-derived, not caller-supplied; removes reference a prior add by commit
ordinal (== table version), resolved by reading _delta_log. The DeltaHarness SPI
gains commitWithData to hide delta-spark behind a neutral CommitRequest.

Review fixes folded in:
- replay schema handling reuses Spark's StructType.toDDL/DataType.fromJson instead
  of a hand-rolled type printer (lossless decimal/nested/quoting; deletes ~45 LOC)
- enforce commit-index == table-version at the commitOp that establishes it, plus a
  non-empty-commit guard, so desync fails at the offending op
- columnMapping is credited to the reader axis only when reader>=2 AND writer>=5
  jointly (mirrors delta-spark implicitlySupportedFeatures)
- read-only specs with no captured table now error instead of silently passing;
  per-spec errors keep the full throwable; cleanup narrows to NonFatal
- document the write spec type and DSL (README, harness/authoring guides); fix
  stale scaladoc

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
…t the source

Three changes in one theme — remove checks that only re-assert what the engine
already guarantees, and move invariant enforcement to where the invariant is
established:

- Delete the write-derived snapshot structural comparison
  (validateSnapshotStructurally) and its helper cluster (declaredConfiguration,
  configurationViolation, normalizedSchema, stripColumnMapping, canonicalField).
  WorkloadValidator replays via Spark from the same commits Spark captured, so a
  table's protocol/metadata/schema is Spark's deterministic output; re-comparing
  it only asserts Spark agrees with itself. Write-derived snapshots are validated
  at capture time and trusted standalone. WorkloadValidator is now
  orchestration-only; snapshot comparison lives solely in SnapshotCapture.
- Delete the protocol-capability check (protocolViolation + the version->implied
  -feature map) for the same reason; it also re-encoded delta-spark's protocol
  rules by hand and got the columnMapping rule wrong.
- Enforce commit-index == table-version inside commitOp (the op that establishes
  it) and guard UpdatePropertiesCommit to set XOR remove (SQL replays SET and
  UNSET as separate commits, which would break the invariant).

validateExpectedLatest now throws-with-context instead of accumulating into a
"warnings" buffer. Row comparison is the only corpus check that validates the
spec rather than Spark's determinism. Deletes the now-obsolete
WorkloadValidatorUnitSuite (it only tested the removed helpers).

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
The log/ package (CommitLog + the typed Action ADT) was dead except for its own
test, while three places hand-rolled the same raw-JSON manipulation it provides.
Make it load-bearing:

- modifyCommitActions now delegates to CommitLog.mutate and hands the modifier
  typed Actions (AddFile, Metadata, ...) instead of (String, ObjectNode) pairs.
  The ~18-line parse/wrap/IO body collapses to a 4-line delegation; callers
  filter by type and `.copy` fields (AddFile.stats stays Option[String], so the
  stats-corruption tests still express empty/garbage stats; commitInfo/cdc
  surface as RawAction and pass through verbatim). Byte-level corruption still
  uses mutateTable directly.
- TableInfoWriter counts removes via CommitLog.read + isInstanceOf[RemoveFile]
  instead of the fragile startsWith("{\"remove") substring match.
- Migrated all 8 modifyCommitActions call sites (CorruptionSuite,
  DataSkippingSuite, WorkloadGeneratorSuite) to the typed modifier.

WriteCommit (spec) / Action (raw log) / CommitRequest (SPI) remain deliberately
separate: different abstractions, no shared information.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
WorkloadGenerator.scala held the generation engine, the DSL state machine, the
handles, and the test-facing facade in one 938-line file. Split by abstraction
(pure move, no behavior change):

- WorkloadGenerator.scala: the generation engine (generateTable) plus the
  internal model it consumes (TableSpec, ReadSpecConfig, SnapshotSpecConfig,
  HasAssertion, WorkloadResult).
- WorkloadContext.scala: the DSL state machine (WorkloadContext + companion) and
  the handles (SpecRef, TableHandle, WriteHandle).
- WorkloadOps.scala: the facade trait mixed into WorkloadTestSuite.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
validateAll walked test dirs sequentially. Each validateTestDir is independent
(its own temp replay tables, path-based table refs, no shared catalog names) and
SparkSession is thread-safe, so run them on a bounded thread pool and let Spark's
scheduler interleave the per-dir jobs. Commits are per-table, so the global
DeltaLog cache only thrashes; the result is unchanged.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
Comment etiquette pass over the files this change touches: replace em-dashes with
plain punctuation (colon/semicolon/period). Comments only; no behavior change.
Pre-existing untouched files and data literals are left alone.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
…n limit

writeExpectedData skips materializing expected_data when the row count exceeds
MaxExpectedDataRows (5M), but validateFromSpec still required the directory,
so any large-table read spec (e.g. DeletionVectorsSuite.DV-017, ~2.1B rows)
failed with "expected_data is missing". When the directory is absent, verify the
re-read row count against the captured count instead; a missing dir at or under
the limit remains a hard failure (incomplete corpus).

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
…y, simplify)

Post-sweep review batch:
- SpecLayout.addPathsAt now resolves remove targets via CommitLog.read + typed
  AddFile instead of hand-parsing commit JSON (the one reimplementation the
  log/ unification missed; robust to JSON key ordering).
- CommitLog.mutate doc now states honestly that it re-serializes through the
  typed model and is not byte-exact (matches behavior; use the raw mutateTable
  path for byte-level corruption).
- Docs: fix the WorkloadValidator scaladoc that still described the deleted
  capability/normalized-schema validation; rewrite the authoring-guide
  modifyCommitActions examples to the typed Action API (the old ones no longer
  compiled); add WorkloadContext/WorkloadValidator to the README components;
  voice fix in spec-reference.
- Simplify pass: drop a dead tuple element in ReadCapture; extract
  partitionedByClause/tblPropertiesClause helpers in WorkloadContext; collapse
  the duplicated read-only validation into a validateReadOnly helper; fix
  TableInfoWriter indentation.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
…mn mapping

replayEvolveSchema rebuilt the StructType by hand and committed a raw
schemaString swap via a metadata-only update. On a non-column-mapping table a
rename/drop done that way reinterprets the physical Parquet columns by the new
logical name, minting a logically-corrupt table. Replay add/rename/drop through
ALTER TABLE instead (mirroring the capture-side ops), so Delta enforces the same
column-mapping requirements it does at capture and rejects an unsafe rename/drop.
Each EvolveSchemaCommit carries exactly one of add/rename/drop, so this still
emits exactly one commit (commit-index == version preserved).

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
The "WriteCommit Fields" table read like the pre-refactor monolithic type (a flat
union of all fields). Reframe it as a field glossary over the per-operation case
classes (the model is a sealed set discriminated by `operation`; each op carries
only its own fields, per the tables above). Also add the missing AppTxn and
AddDomainMetadata value-type tables, which were referenced but never defined.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
… test rows

Loosen the row parameter of insertOp/replaceTableOp from Seq to
IterableOnce[Map[String, Any]] (materialized once internally), so callers can
pass a range/iterator/view, not just a literal Seq. All existing Seq(...) callers
still compile.

Convert the genuinely formulaic insert blocks in WriteBasicSuite to ranges
(delete_basic, category grouping, value=id*100); left the irregular ones
(named rows, per-type values) as explicit Seq, where a range would be less clear.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
Convert multi-line `Seq(Map(...))` insert/replace row lists in WriteBasicSuite
and WriteSequencesSuite to compact range/zipWithIndex iterators. Generated data
is byte-identical (verified: WORKLOAD_FORCE=true suites pass). Single-row inserts
and the per-type rows in WriteTypesSuite are left explicit (no LOC win there).
Net -28 lines.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
The write-derived snapshot branch had become a no-op (replay-or-throw only),
so a whole class of spec validated nothing. Restore real validation: compare the
replayed table's snapshot against the captured spec — protocol exact-equality
(replay is Spark from the same commits, deterministic), declared-config keys,
column-mapping-normalized schemaString, and partitionColumns/format. Table id and
createdTime differ per table instance and are intentionally not compared.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
Delete the standalone RowParquet object; add schemaAt / writeRows /
writeRowsToTemp to the DeltaHarness trait (implemented in DeltaSparkHarness, with
the value coercion kept private and unchanged). Rewire the capture-side callers
(WorkloadContext, WriteSpecCapture) to the harness. Consolidates this Spark-only
capture logic behind the SPI and removes a file.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
A prior cleanup reindented the whole write() body (~200 whitespace-only diff
lines). Restore the original indentation, keeping only the real change (typed
remove-count via CommitLog.read).

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
…record()

Replace the 11 record* wrappers with a single `record(commit, rows): Int` (append
+ stage rows + return ordinal); callers in WorkloadContext now build the typed ADT
inline. Keep only the genuine converters (ddlToSchemaJson, addColumnsJson) and
recordLowLevelRows. Also drop dead code in WorkloadTestSuite. Net ~-64 lines; the
SET-XOR-UNSET invariant is preserved structurally and still enforced at replay.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
… split)

Fold WorkloadContext and WorkloadOps back into WorkloadGenerator.scala so PR #55
is the write-generator feature, not a file reorganization. Pure code movement (no
behavior change); the split will land as its own follow-up PR. Cuts ~1k churn
lines from the PR.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
Delete the obsolete WriteCommit field glossary (per-op tables already cover it),
merge the four low-level value-type tables into one, and dedup the examples
(8 -> 4). Net ~-117 lines; content stays accurate to the ADT.

Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
Replace the two index-keyed side maps (rowData, lowLevelRows) joined to the
commits buffer by ordinal with a single buffer of WriteOpWithData(commit, rows).
Rows now travel with the commit; buildSpec reads them by destructuring instead
of HashMap lookups. Drops recordLowLevelRows and the implicit "same int" join.

Co-authored-by: Isaac
…mit()

The harness SPI had a dead no-data commit() and a commitWithData() taking data
files as a separate param. Make data files part of CommitRequest (addDataParquet)
and expose a single commit(spark, tablePath, req): Seq[String]. Also trim the
AddFileInput doc to the two non-obvious facts.

Co-authored-by: Isaac
OussamaSaoudi-db and others added 2 commits June 22, 2026 23:04
…ition, trim writeExpectedLatest

- writeRowsToTemp was pure sugar over writeRows; move it into WorkloadContext (its only callers)
  so the DeltaHarness SPI exposes just the writeRows primitive.
- schemaAt's includePartition param + its false branch were dead (every caller passed true); remove.
- writeExpectedLatest wrote a table_version_metadata.json that nothing reads; keep only table_content.

Co-authored-by: Isaac
@rliao147
rliao147 force-pushed the stack/add-write-generator branch from 7393511 to 6a827a5 Compare June 22, 2026 23:11
@rliao147
rliao147 removed the request for review from nicklan June 22, 2026 23:11
@rliao147
rliao147 force-pushed the stack/add-write-generator branch from 40d75cf to 40c1bad Compare June 23, 2026 20:52
@rliao147 rliao147 closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants