Repository navigation
Conversation
Collaborator
Author
|
Recommended review order:
|
| | `delete` | `predicate` | Delete rows matching predicate | | ||
| | `evolve_schema` | `addColumns?`, `renameColumns?`, `dropColumns?` | Modify table schema | | ||
| | `update_properties` | `set?`, `remove?` | Modify table properties | | ||
| | `restore` | `version` | Restore table to a previous version | |
Collaborator
There was a problem hiding this comment.
might be worth removing restore for now since I don't think it's really in demand, and this reference validator asterisk is a little concerning.
|
|
||
| | Operation | Fields | Description | | ||
| |-----------|--------|-------------| | ||
| | `create_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Create a new table with the given schema | |
Collaborator
There was a problem hiding this comment.
We should definitely include Replace tale. That has been the trickiest, yuckiest part of table formats and will be super useful for us to get the semantics right.
Collaborator
There was a problem hiding this comment.
Could be done as a followup, but given that we're doing it all in one go, I'd prefer we do it eagerly if we can.
| |-----------|--------|-------------| | ||
| | `create_table` | `schema`, `partitionColumns?`, `properties?`, `dataFiles?` | Create a new table with the given schema | | ||
| | `insert` | `dataFiles?` | Append rows from data files | | ||
| | `update` | `predicate`, `set` | Update rows matching predicate | |
Collaborator
There was a problem hiding this comment.
Suggested change
| | `update` | `predicate`, `set` | Update rows matching predicate | | |
| | `update` | `predicate`, `set` | For each row matching the predicate, set it to at value | |
^ Something like thi smight be more precise
OussamaSaoudi
pushed a commit
that referenced
this pull request
Jun 17, 2026
… split) Fold WorkloadContext and WorkloadOps back into WorkloadGenerator.scala so PR #55 is the write-generator feature, not a file reorganization. Pure code movement (no behavior change); the split will land as its own follow-up PR. Cuts ~1k churn lines from the PR. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
This PR adds a write spec, which allows DAT workloads to test Delta writers. Any Delta writer can validate their own write APIs through the existing read/snapshot specs.
…l commits via the engine Make "everything a spec": one WorkloadValidator validates read/snapshot/write, dispatching on `type`. Read/snapshot specs carry an optional `writeSpec` pointer marking them write-derived (reconstruct the table by replaying the sibling write spec, then validate portably) vs read-only (validate the captured delta/ exactly). Deletes the separate WriteSpecValidator. Model the write spec as a sealed WriteCommit ADT (one *Commit case class per op), adding replace_table (incl. replace-as-select) and removing restore. Low-level `commit` adds carry logical rows materialized to Parquet and written through the engine's OptimisticTransaction.writeFiles, so column mapping / partitioning / stats are engine-derived, not caller-supplied; removes reference a prior add by commit ordinal (== table version), resolved by reading _delta_log. The DeltaHarness SPI gains commitWithData to hide delta-spark behind a neutral CommitRequest. Review fixes folded in: - replay schema handling reuses Spark's StructType.toDDL/DataType.fromJson instead of a hand-rolled type printer (lossless decimal/nested/quoting; deletes ~45 LOC) - enforce commit-index == table-version at the commitOp that establishes it, plus a non-empty-commit guard, so desync fails at the offending op - columnMapping is credited to the reader axis only when reader>=2 AND writer>=5 jointly (mirrors delta-spark implicitlySupportedFeatures) - read-only specs with no captured table now error instead of silently passing; per-spec errors keep the full throwable; cleanup narrows to NonFatal - document the write spec type and DSL (README, harness/authoring guides); fix stale scaladoc Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
…t the source Three changes in one theme — remove checks that only re-assert what the engine already guarantees, and move invariant enforcement to where the invariant is established: - Delete the write-derived snapshot structural comparison (validateSnapshotStructurally) and its helper cluster (declaredConfiguration, configurationViolation, normalizedSchema, stripColumnMapping, canonicalField). WorkloadValidator replays via Spark from the same commits Spark captured, so a table's protocol/metadata/schema is Spark's deterministic output; re-comparing it only asserts Spark agrees with itself. Write-derived snapshots are validated at capture time and trusted standalone. WorkloadValidator is now orchestration-only; snapshot comparison lives solely in SnapshotCapture. - Delete the protocol-capability check (protocolViolation + the version->implied -feature map) for the same reason; it also re-encoded delta-spark's protocol rules by hand and got the columnMapping rule wrong. - Enforce commit-index == table-version inside commitOp (the op that establishes it) and guard UpdatePropertiesCommit to set XOR remove (SQL replays SET and UNSET as separate commits, which would break the invariant). validateExpectedLatest now throws-with-context instead of accumulating into a "warnings" buffer. Row comparison is the only corpus check that validates the spec rather than Spark's determinism. Deletes the now-obsolete WorkloadValidatorUnitSuite (it only tested the removed helpers). Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
The log/ package (CommitLog + the typed Action ADT) was dead except for its own
test, while three places hand-rolled the same raw-JSON manipulation it provides.
Make it load-bearing:
- modifyCommitActions now delegates to CommitLog.mutate and hands the modifier
typed Actions (AddFile, Metadata, ...) instead of (String, ObjectNode) pairs.
The ~18-line parse/wrap/IO body collapses to a 4-line delegation; callers
filter by type and `.copy` fields (AddFile.stats stays Option[String], so the
stats-corruption tests still express empty/garbage stats; commitInfo/cdc
surface as RawAction and pass through verbatim). Byte-level corruption still
uses mutateTable directly.
- TableInfoWriter counts removes via CommitLog.read + isInstanceOf[RemoveFile]
instead of the fragile startsWith("{\"remove") substring match.
- Migrated all 8 modifyCommitActions call sites (CorruptionSuite,
DataSkippingSuite, WorkloadGeneratorSuite) to the typed modifier.
WriteCommit (spec) / Action (raw log) / CommitRequest (SPI) remain deliberately
separate: different abstractions, no shared information.
Co-authored-by: Ryan Liao <ryanliao123@gmail.com>
Co-authored-by: Isaac
WorkloadGenerator.scala held the generation engine, the DSL state machine, the handles, and the test-facing facade in one 938-line file. Split by abstraction (pure move, no behavior change): - WorkloadGenerator.scala: the generation engine (generateTable) plus the internal model it consumes (TableSpec, ReadSpecConfig, SnapshotSpecConfig, HasAssertion, WorkloadResult). - WorkloadContext.scala: the DSL state machine (WorkloadContext + companion) and the handles (SpecRef, TableHandle, WriteHandle). - WorkloadOps.scala: the facade trait mixed into WorkloadTestSuite. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
validateAll walked test dirs sequentially. Each validateTestDir is independent (its own temp replay tables, path-based table refs, no shared catalog names) and SparkSession is thread-safe, so run them on a bounded thread pool and let Spark's scheduler interleave the per-dir jobs. Commits are per-table, so the global DeltaLog cache only thrashes; the result is unchanged. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
Comment etiquette pass over the files this change touches: replace em-dashes with plain punctuation (colon/semicolon/period). Comments only; no behavior change. Pre-existing untouched files and data literals are left alone. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
…n limit writeExpectedData skips materializing expected_data when the row count exceeds MaxExpectedDataRows (5M), but validateFromSpec still required the directory, so any large-table read spec (e.g. DeletionVectorsSuite.DV-017, ~2.1B rows) failed with "expected_data is missing". When the directory is absent, verify the re-read row count against the captured count instead; a missing dir at or under the limit remains a hard failure (incomplete corpus). Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
…y, simplify) Post-sweep review batch: - SpecLayout.addPathsAt now resolves remove targets via CommitLog.read + typed AddFile instead of hand-parsing commit JSON (the one reimplementation the log/ unification missed; robust to JSON key ordering). - CommitLog.mutate doc now states honestly that it re-serializes through the typed model and is not byte-exact (matches behavior; use the raw mutateTable path for byte-level corruption). - Docs: fix the WorkloadValidator scaladoc that still described the deleted capability/normalized-schema validation; rewrite the authoring-guide modifyCommitActions examples to the typed Action API (the old ones no longer compiled); add WorkloadContext/WorkloadValidator to the README components; voice fix in spec-reference. - Simplify pass: drop a dead tuple element in ReadCapture; extract partitionedByClause/tblPropertiesClause helpers in WorkloadContext; collapse the duplicated read-only validation into a validateReadOnly helper; fix TableInfoWriter indentation. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
…mn mapping replayEvolveSchema rebuilt the StructType by hand and committed a raw schemaString swap via a metadata-only update. On a non-column-mapping table a rename/drop done that way reinterprets the physical Parquet columns by the new logical name, minting a logically-corrupt table. Replay add/rename/drop through ALTER TABLE instead (mirroring the capture-side ops), so Delta enforces the same column-mapping requirements it does at capture and rejects an unsafe rename/drop. Each EvolveSchemaCommit carries exactly one of add/rename/drop, so this still emits exactly one commit (commit-index == version preserved). Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
The "WriteCommit Fields" table read like the pre-refactor monolithic type (a flat union of all fields). Reframe it as a field glossary over the per-operation case classes (the model is a sealed set discriminated by `operation`; each op carries only its own fields, per the tables above). Also add the missing AppTxn and AddDomainMetadata value-type tables, which were referenced but never defined. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
… test rows Loosen the row parameter of insertOp/replaceTableOp from Seq to IterableOnce[Map[String, Any]] (materialized once internally), so callers can pass a range/iterator/view, not just a literal Seq. All existing Seq(...) callers still compile. Convert the genuinely formulaic insert blocks in WriteBasicSuite to ranges (delete_basic, category grouping, value=id*100); left the irregular ones (named rows, per-type values) as explicit Seq, where a range would be less clear. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
Convert multi-line `Seq(Map(...))` insert/replace row lists in WriteBasicSuite and WriteSequencesSuite to compact range/zipWithIndex iterators. Generated data is byte-identical (verified: WORKLOAD_FORCE=true suites pass). Single-row inserts and the per-type rows in WriteTypesSuite are left explicit (no LOC win there). Net -28 lines. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
The write-derived snapshot branch had become a no-op (replay-or-throw only), so a whole class of spec validated nothing. Restore real validation: compare the replayed table's snapshot against the captured spec — protocol exact-equality (replay is Spark from the same commits, deterministic), declared-config keys, column-mapping-normalized schemaString, and partitionColumns/format. Table id and createdTime differ per table instance and are intentionally not compared. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
Delete the standalone RowParquet object; add schemaAt / writeRows / writeRowsToTemp to the DeltaHarness trait (implemented in DeltaSparkHarness, with the value coercion kept private and unchanged). Rewire the capture-side callers (WorkloadContext, WriteSpecCapture) to the harness. Consolidates this Spark-only capture logic behind the SPI and removes a file. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
A prior cleanup reindented the whole write() body (~200 whitespace-only diff lines). Restore the original indentation, keeping only the real change (typed remove-count via CommitLog.read). Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
…record() Replace the 11 record* wrappers with a single `record(commit, rows): Int` (append + stage rows + return ordinal); callers in WorkloadContext now build the typed ADT inline. Keep only the genuine converters (ddlToSchemaJson, addColumnsJson) and recordLowLevelRows. Also drop dead code in WorkloadTestSuite. Net ~-64 lines; the SET-XOR-UNSET invariant is preserved structurally and still enforced at replay. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
… split) Fold WorkloadContext and WorkloadOps back into WorkloadGenerator.scala so PR #55 is the write-generator feature, not a file reorganization. Pure code movement (no behavior change); the split will land as its own follow-up PR. Cuts ~1k churn lines from the PR. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
Delete the obsolete WriteCommit field glossary (per-op tables already cover it), merge the four low-level value-type tables into one, and dedup the examples (8 -> 4). Net ~-117 lines; content stays accurate to the ADT. Co-authored-by: Ryan Liao <ryanliao123@gmail.com> Co-authored-by: Isaac
Replace the two index-keyed side maps (rowData, lowLevelRows) joined to the commits buffer by ordinal with a single buffer of WriteOpWithData(commit, rows). Rows now travel with the commit; buildSpec reads them by destructuring instead of HashMap lookups. Drops recordLowLevelRows and the implicit "same int" join. Co-authored-by: Isaac
…mit() The harness SPI had a dead no-data commit() and a commitWithData() taking data files as a separate param. Make data files part of CommitRequest (addDataParquet) and expose a single commit(spark, tablePath, req): Seq[String]. Also trim the AddFileInput doc to the two non-obvious facts. Co-authored-by: Isaac
…ition, trim writeExpectedLatest - writeRowsToTemp was pure sugar over writeRows; move it into WorkloadContext (its only callers) so the DeltaHarness SPI exposes just the writeRows primitive. - schemaAt's includePartition param + its false branch were dead (every caller passed true); remove. - writeExpectedLatest wrote a table_version_metadata.json that nothing reads; keep only table_content. Co-authored-by: Isaac
rliao147
force-pushed
the
stack/add-write-generator
branch
from
June 22, 2026 23:11
7393511 to
6a827a5
Compare
rliao147
force-pushed
the
stack/add-write-generator
branch
from
June 23, 2026 20:52
40d75cf to
40c1bad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.
Summary
DAT workloads only test readers (
read/snapshotspecs). This PR adds awritespec:
write_spec.jsoncontains commits that any Delta writer can replay through its own write API, then validate against existing read/snapshot specs.create_table,insert,update,delete, add-column schema evolution, set/unset properties,restore) andcommitfor raw add/remove-file,SetTransaction, andDomainMetadataactions.WriteSpecCapturerecords each op and copies the data files it producesWriteSpecValidatorreplays every commit inwrite_spec.jsoninto a fresh table and verifies the final table against expected/latest.Co-author credit to @OussamaSaoudi since most of the code was ported over from his local fork.