Repository navigation
feat: add cdf, checkpoint, and crc spec types - #56
Conversation
95f9bbb to
88cfc7d
Compare
9429be6 to
cd75797
Compare
6674492 to
31ad828
Compare
ea920a2 to
f29e717
Compare
f29e717 to
8deeb01
Compare
9f707d3 to
d6b2960
Compare
d6b2960 to
10e9ebe
Compare
| require(checkpointFileExists(tablePath, spec.version), | ||
| s"Checkpoint validation failed for $specName: no checkpoint file at version ${spec.version}; " + | ||
| "the engine under test did not write one") |
There was a problem hiding this comment.
Can we do any stronger assertions here? Ex: could the harness let us read the checkpoint contents at a version and compare it against expected contents? At least for read-only tables, we should be able to do a pretty strong assertion here for domain metadata, protocol, metadata, app txn, add files, and remove files.
There was a problem hiding this comment.
I would prefer to rely on readSpec, snapshotSpec for metadata validations for now. If you feel strongly about this, could I add it as a follow up change? The problem with adding strong assertions here is that the harness will have to implement their own parquet/json handlers to perform these validations. But you bring up a good point that readSpec/SnapshotSpec validations won't be enough if the harness doesn't actually read through the checkpoint/CRC.
There was a problem hiding this comment.
cool, let's do followup.
For read-only specs, the biggest difference is gonna be checks on apptxn, domain metadata, and file tombstones. I also think of each spec as its own test case, so I'd want the "expected data" to have at least something for us to check against.
🥞 Stacked PR
Use this link to review incremental changes.
Description
This change adds three validation spec types to the workload generator so a harness can assert an engine's checkpoint, version-checksum (CRC), and change-data-feed output.
How was this patch tested?
sbt "testOnly io.delta.workload.tables.*" generates and self-validates the new cases.