Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions workload-generator/docs/harness-implementation-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,34 @@ The write spec has no expected-data artifact of its own. Its own check is **basi

Because a write spec is replayed (not byte-compared), comparison is **portable**: rows-only data checks, capability-based protocol comparison (your table may use explicit table features where Spark used legacy versions), and column-mapping-normalized schema. A write spec's presence makes its whole directory *write-derived*: replay it into a fresh table once, then validate every read/snapshot spec in that directory (including the baseline `latest` read) against the replay (portably), rather than against a captured `delta/` directory.

### CDF Specs

```json
{ "type": "cdf", "startVersion": 0, "endVersion": 2, "expected": { "rowCount": 5 } }
```

**Execute:** read the change feed over `[startVersion, endVersion]` (or the timestamp bounds).

**Validate:** compare the change rows against `expected/<name>/expected_data/` as a bag of typed rows, exactly like a read spec. `_change_type` and `_commit_version` are compared; `_commit_timestamp` is not (on non-ICT tables it is the commit file's mtime, which does not survive tarring). Assert `rowCount`. An `"error"` outcome is supported like other specs.

### Checkpoint and CRC Specs

```json
{ "type": "checkpoint", "version": 2 }
{ "type": "crc", "version": 2 }
```

These are **presence triggers**, not content specs. Validate by asserting a `<version>.checkpoint*` (checkpoint) or `<version>.crc` (crc) file exists in `_delta_log/`. Checkpoint contents are validated indirectly by a paired read spec: reading a version whose snapshot reconstructs through the checkpoint returns wrong rows if the checkpoint dropped or duplicated an Add file, so a content-wrong checkpoint fails that read spec. (A snapshot spec covers only protocol/metadata, not the file set, so it does not catch this on its own.) The `.crc` (version-checksum) file is optional per the protocol, so the crc spec is capability-gated: a harness whose engine does not write version checksums skips crc specs, while a harness whose engine does write them must produce a `.crc` at the spec's version. The spec asserts only that the file was written; an engine that verifies checksums on read additionally validates the contents on load, but that is not assumed.

### Invariant Validators (row tracking, stats)

Some correctness properties are not per-spec JSON; they are engine-agnostic invariants a write harness checks against the table the engine under test produced. A write harness implements these directly against the written table:

- **Stats:** per-file timestamp stats are millisecond-truncated; each file's `minValues`/`maxValues` bound its live rows (deletion vectors applied). When `tightBounds` is true the bounds must equal the live min/max; when false they need only contain them.
- **Row tracking (when the `rowTracking` feature is on):** every live Add carries a `baseRowId`; fresh-id ranges never overlap; the `rowIdHighWaterMark` covers the largest assigned id; and every row's `_metadata.row_id` is non-null and unique.

These run only in a write harness, against the table the engine under test produced.

---

## Step 4: Incremental Adoption
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,9 @@ class SpecRef[T] private[workload] (

private[workload] trait HasAssertion[T] {
var assertion: Option[com.fasterxml.jackson.databind.JsonNode => Unit] = None
def deserialize: com.fasterxml.jackson.databind.JsonNode => T
def specClass: Class[T]
def deserialize: com.fasterxml.jackson.databind.JsonNode => T =
n => JsonUtil.mapper.treeToValue(n, specClass)
}

// ---------------------------------------------------------------------------
Expand All @@ -76,20 +78,39 @@ private[workload] class TableDecl(
private[workload] def resolveOutputName(name: String): Unit = { _outputName = name }
val readSpecs = mutable.ArrayBuffer[ReadSpecConfig]()
val snapshotSpecs = mutable.ArrayBuffer[SnapshotSpecConfig]()
val cdfSpecs = mutable.ArrayBuffer[CdfSpecConfig]()
val checkpointSpecs = mutable.ArrayBuffer[CheckpointSpecConfig]()
val crcSpecs = mutable.ArrayBuffer[CrcSpecConfig]()
val mutations = mutable.ArrayBuffer[Path => Unit]()
var writeBuilder: Option[WriteSpecBuilder] = None
}

private[workload] case class ReadSpecConfig(
name: String, query: ReadQuery,
expectError: ErrorExpectation = AutoDetect) extends HasAssertion[ReadSpec] {
val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) =>
JsonUtil.mapper.treeToValue(n, classOf[ReadSpec])
val specClass = classOf[ReadSpec]
}

private[workload] case class SnapshotSpecConfig(
query: SnapshotQuery,
expectError: ErrorExpectation = AutoDetect) extends HasAssertion[SnapshotSpec] {
val deserialize = (n: com.fasterxml.jackson.databind.JsonNode) =>
JsonUtil.mapper.treeToValue(n, classOf[SnapshotSpec])
val specClass = classOf[SnapshotSpec]
}

private[workload] case class CdfSpecConfig(
name: String, startVersion: Option[Long], endVersion: Option[Long],
startTimestamp: Option[String], endTimestamp: Option[String],
predicate: Option[String], columns: Option[Seq[String]],
expectError: Option[String] = None) extends HasAssertion[CdfSpec] {
val specClass = classOf[CdfSpec]
}

private[workload] case class CheckpointSpecConfig(
name: String, version: Long) extends HasAssertion[CheckpointSpec] {
val specClass = classOf[CheckpointSpec]
}

private[workload] case class CrcSpecConfig(
name: String, version: Long) extends HasAssertion[CrcSpec] {
val specClass = classOf[CrcSpec]
}
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,44 @@ class WorkloadContext private[workload] (
if (parts.length == 1) "read_all" else parts.mkString("_")
}

private[workload] def autoCdfName(
startVersion: Option[Long],
endVersion: Option[Long],
startTimestamp: Option[String],
endTimestamp: Option[String],
predicate: Option[String],
columns: Option[Seq[String]]): String = {
val parts = mutable.ArrayBuffer[String]("cdf")
startVersion.foreach(v => parts += s"v$v")
startTimestamp.foreach(ts => parts += "ts_" + ts.replace(":", "-").replace(" ", "_").take(23))
parts += "to"
endVersion.foreach(v => parts += s"v$v")
endTimestamp.foreach(ts => parts += "ts_" + ts.replace(":", "-").replace(" ", "_").take(23))
if (endVersion.isEmpty && endTimestamp.isEmpty) parts += "latest"
predicate.foreach(p => parts += simplifyPredicate(p))
columns.foreach { cols =>
parts += "cols_" + cols.take(3).mkString("_")
if (cols.size > 3) parts += s"plus${cols.size - 3}"
}
parts.mkString("_")
}

/** Turn a SQL predicate into a filesystem-safe spec-name fragment. */
private def simplifyPredicate(p: String): String =
p.toLowerCase
.replaceAll(">=", "_gte_").replaceAll("<=", "_lte_")
.replaceAll("<>", "_neq_").replaceAll("!=", "_neq_")
.replaceAll(">", "_gt_").replaceAll("<", "_lt_")
.replaceAll("=", "_eq_")
.replaceAll("\\s+and\\s+", "_and_").replaceAll("\\s+or\\s+", "_or_")
.replaceAll("\\s+is\\s+not\\s+null", "_is_not_null")
.replaceAll("\\s+is\\s+null", "_is_null")
.replaceAll("\\s+in\\s+", "_in_")
.replaceAll("\\s+", "_")
.replaceAll("[^a-z0-9_]", "")
.replaceAll("_+", "_").stripPrefix("_").stripSuffix("_")
.take(50)

// ---- Internal ----

private def ensureTableSpec(handle: TableHandle): Unit = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import scala.collection.mutable
import org.apache.commons.io.FileUtils
import org.apache.spark.sql.SparkSession

import io.delta.workload.capture.{ReadCapture, SnapshotCapture, TableInfoCapture}
import io.delta.workload.capture.{CdfCapture, CheckpointCapture, CrcCapture, ReadCapture, SnapshotCapture, TableInfoCapture}
import io.delta.workload.json.JsonUtil
import io.delta.workload.model._
import io.delta.workload.validate.WorkloadValidator
Expand Down Expand Up @@ -97,16 +97,11 @@ object WorkloadGenerator {
Seq(SnapshotSpecConfig(SnapshotQuery()))
} else ts.snapshotSpecs
for (ss <- explicits) {
SnapshotCapture.capture(spark, dirName, destTablePath, specsDir,
val specPath = SnapshotCapture.capture(spark, dirName, destTablePath, specsDir,
query = ss.query, expectError = ss.expectError)
val specName = (ss.query.version, ss.query.timestamp) match {
case (Some(v), _) => s"${dirName}_snapshot_v$v"
case (_, Some(ts)) =>
s"${dirName}_snapshot_ts_${ts.replace(":", "-").replace(" ", "_")}"
case _ => s"${dirName}_snapshot"
}
val specName = specPath.getFileName.toString.stripSuffix(".json")
if (!snapshotNames.contains(specName)) snapshotNames += specName
checkAssertion(ss, specsDir.resolve(s"$specName.json"))
checkAssertion(ss, specPath)
}

val readNames = mutable.ArrayBuffer[String]()
Expand All @@ -131,6 +126,33 @@ object WorkloadGenerator {
builder.buildSpec(spark, destTablePath, testOutputDir)
}

// Running count of all specs generated for this case.
var totalSpecCount = 0

for (cs <- ts.cdfSpecs) {
val specPath = CdfCapture.capture(spark, dirName, destTablePath, testOutputDir, specsDir,
name = cs.name, startVersion = cs.startVersion, endVersion = cs.endVersion,
startTimestamp = cs.startTimestamp, endTimestamp = cs.endTimestamp,
predicate = cs.predicate, columns = cs.columns, expectError = cs.expectError)
totalSpecCount += 1
checkAssertion(cs, specPath)
}

for (cp <- ts.checkpointSpecs) {
val specPath = CheckpointCapture.capture(spark, dirName, destTablePath, specsDir,
version = cp.version, name = cp.name)
totalSpecCount += 1
checkAssertion(cp, specPath)
}

for (crc <- ts.crcSpecs) {
CrcCapture.capture(spark, dirName, destTablePath, specsDir,
version = crc.version, name = crc.name).foreach { p =>
totalSpecCount += 1
checkAssertion(crc, p)
}
}

// table_info.json. An intentionally-mutated table (corruption / deleted-log tests) may be
// unreadable and so can't produce it; an unmutated table that fails here is a real bug, so
// only a mutated table tolerates the failure.
Expand All @@ -156,8 +178,8 @@ object WorkloadGenerator {
Files.write(reproDir.resolve("generate.scala"),
s"// Generated by ${ts.outputName} test\n".getBytes("UTF-8"))

val total = snapshotNames.size + readNames.size
println(s" $dirName: $total specs")
totalSpecCount += snapshotNames.size + readNames.size
println(s" $dirName: $totalSpecCount specs")

dirName
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,62 @@ trait WorkloadOps {
new SpecRef(config)
}

/**
* Change Data Feed spec over a version/timestamp range. Requires a start bound
* (`startVersion` or `startTimestamp`).
*/
def cdfSpec(
table: TableHandle,
startVersion: java.lang.Long = null,
endVersion: java.lang.Long = null,
startTimestamp: String = null,
endTimestamp: String = null,
predicate: String = null,
columns: Seq[String] = null,
name: String = null,
expectError: String = null): SpecRef[CdfSpec] = {
val ctx = WorkloadContext.current
val specName = if (name != null) name else ctx.autoCdfName(
ctx.toOption(startVersion), ctx.toOption(endVersion), Option(startTimestamp),
Option(endTimestamp), Option(predicate), Option(columns))
ctx.requireUnique(table, specName)
val config = CdfSpecConfig(
specName, ctx.toOption(startVersion), ctx.toOption(endVersion), Option(startTimestamp),
Option(endTimestamp), Option(predicate), Option(columns), Option(expectError))
ctx.getTableSpec(table).cdfSpecs += config
new SpecRef(config)
}

/**
* Checkpoint spec: a trigger that forces a checkpoint at `version`.
*/
def checkpointSpec(
table: TableHandle,
version: Long,
name: String = null): SpecRef[CheckpointSpec] = {
val ctx = WorkloadContext.current
val specName = if (name != null) name else s"checkpoint_v$version"
ctx.requireUnique(table, specName)
val config = CheckpointSpecConfig(specName, version)
ctx.getTableSpec(table).checkpointSpecs += config
new SpecRef(config)
}

/**
* CRC (version-checksum) spec: a trigger asserting the engine wrote a `<version>.crc` at `version`.
*/
def crcSpec(
table: TableHandle,
version: Long,
name: String = null): SpecRef[CrcSpec] = {
val ctx = WorkloadContext.current
val specName = if (name != null) name else s"crc_v$version"
ctx.requireUnique(table, specName)
val config = CrcSpecConfig(specName, version)
ctx.getTableSpec(table).crcSpecs += config
new SpecRef(config)
}

/** Force Spark to write a checkpoint file for the given SQL table name. */
def forceCheckpoint(tableName: String): Unit = current.forceCheckpoint(tableName)

Expand Down Expand Up @@ -350,4 +406,18 @@ trait WorkloadOps {
if (Files.exists(CommitLog.commitFile(tableDir, version)))
CommitLog.mutate(tableDir, version)(modifier)
}

/**
* Append `actions` to the commit JSON at `version`.
*/
def injectCommitActions(table: TableHandle, version: Long)(actions: Seq[Action]): Unit =
mutateTable(table) { tableDir =>
val logDir = tableDir.resolve("_delta_log")
val commitFile = logDir.resolve(f"$version%020d.json")
val content = new String(Files.readAllBytes(commitFile), "UTF-8")
Files.write(
commitFile,
(content.trim + "\n" + actions.map(_.toJson).mkString("\n") + "\n").getBytes("UTF-8"))
Files.deleteIfExists(logDir.resolve(f"$version%020d.crc"))
}
}
Loading
Loading