Skip to content

feat: add cdf, checkpoint, and crc spec types - #56

Merged
rliao147 merged 3 commits into
mainfrom
stack/add-missing-ops
Sep 23, 2026
Merged

rliao147 merged 3 commits into
mainfrom
stack/add-missing-ops

Conversation

@rliao147

@rliao147 rliao147 commented Jun 16, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


Description

This change adds three validation spec types to the workload generator so a harness can assert an engine's checkpoint, version-checksum (CRC), and change-data-feed output.

  • New spec types: checkpoint, crc, cdf.
  • Capture objects CheckpointCapture, CrcCapture, CdfCapture produce/validate each spec
  • Corpus: new CheckpointsSuite, CrcSuite, CdcSuite (cp_, crc_, cdf_*), plus write-derived checkpoint/crc/cdf cases in WriteMetadataSpecsSuite/WriteCommitSuite.

How was this patch tested?

sbt "testOnly io.delta.workload.tables.*" generates and self-validates the new cases.

@rliao147
rliao147 force-pushed the stack/add-missing-ops branch from 95f9bbb to 88cfc7d Compare June 16, 2026 17:12
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch 3 times, most recently from 9429be6 to cd75797 Compare June 23, 2026 20:52
@rliao147 rliao147 closed this Aug 26, 2026
@rliao147 rliao147 reopened this Aug 26, 2026
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch 3 times, most recently from 6674492 to 31ad828 Compare August 26, 2026 21:01
@rliao147 rliao147 changed the title feat: add cdf, domain_metadata, appTxn, checkpoint, and crc spec types feat: add cdf, checkpoint, and crc spec types Aug 26, 2026
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch 2 times, most recently from ea920a2 to f29e717 Compare August 26, 2026 22:15
@rliao147
rliao147 requested a review from OussamaSaoudi August 26, 2026 22:15
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch from f29e717 to 8deeb01 Compare August 27, 2026 18:50
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch 5 times, most recently from 9f707d3 to d6b2960 Compare September 18, 2026 20:16
@rliao147
rliao147 force-pushed the stack/add-missing-ops branch from d6b2960 to 10e9ebe Compare September 18, 2026 21:36
Comment on lines +57 to +59
require(checkpointFileExists(tablePath, spec.version),
s"Checkpoint validation failed for $specName: no checkpoint file at version ${spec.version}; " +
"the engine under test did not write one")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do any stronger assertions here? Ex: could the harness let us read the checkpoint contents at a version and compare it against expected contents? At least for read-only tables, we should be able to do a pretty strong assertion here for domain metadata, protocol, metadata, app txn, add files, and remove files.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to rely on readSpec, snapshotSpec for metadata validations for now. If you feel strongly about this, could I add it as a follow up change? The problem with adding strong assertions here is that the harness will have to implement their own parquet/json handlers to perform these validations. But you bring up a good point that readSpec/SnapshotSpec validations won't be enough if the harness doesn't actually read through the checkpoint/CRC.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, let's do followup.

For read-only specs, the biggest difference is gonna be checks on apptxn, domain metadata, and file tombstones. I also think of each spec as its own test case, so I'd want the "expected data" to have at least something for us to check against.

@rliao147
rliao147 merged commit 63b0464 into main Sep 23, 2026
1 check passed
@rliao147
rliao147 deleted the stack/add-missing-ops branch September 23, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants