Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,11 @@ jobs:
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version-file: ".python-version"

- name: Install uv
uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
with:
enable-cache: true

- name: Install the project
run: uv sync --locked --dev

- name: Cache pre-commit
uses: actions/cache@0400d5f644dc74513175e3cd8d07132dd4860809 # v4.2.4
with:
Expand All @@ -33,9 +25,3 @@ jobs:

- name: Run pre-commit
run: uvx pre-commit run --show-diff-on-failure --color=always --all-files

- name: Generate tables
run: uv run --no-sync dat generate-tables

- name: Run tests
run: uv run --no-sync pytest tests
1 change: 0 additions & 1 deletion .python-version

This file was deleted.

33 changes: 9 additions & 24 deletions contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,28 @@

## Getting started

This repository uses `poetry` to handle Python dependencies and an isolated environment. To install the environment, run
The workload generator is an sbt project. To compile it, run

```
poetry install
cd workload-generator
sbt Test/compile
```

Most common tasks are defined in the Makefile. For example, to lint the Python files run
Most common tasks are defined in the justfile. For example, to run tests:

```
make lint
just test
```

To open a shell session within the poetry environment, use
## Generating workloads locally

```
poetry shell
```

## Generating reference tables locally

Reader test cases are defined in `dat/generated_tables.py`. Currently, they are all defined in terms of PySpark code. If there is a table that doesn't fit into that paradigm, create an issue on the repository and we will consider other modes of adding tables.

To regenerate tables, run

```
make write-generated-tables
```
Workload suites are defined under `workload-generator/src/test/scala/io/delta/workload/tables`.
See `workload-generator/docs/authoring-guide.md` for authoring and generation instructions.

## Adding writer tests

TBD

## Running unit tests

Note: You need to generate the reference tables locally in order to run the unit tests.

You can run the unit tests with `poetry run pytest tests`.

If you've already run `poetry shell`, you can run the tests with `pytest tests`.

Run `just test`, or use the focused sbt commands documented in `workload-generator/README.md`.
14 changes: 5 additions & 9 deletions justfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# setup the local environment to develop dat
# setup local pre-commit hooks
setup:
uv sync
uvx pre-commit install

# update pre-commit hooks
Expand All @@ -11,12 +10,9 @@ update-hooks:
lint:
uvx pre-commit run --all-files

# run unit tests
# run workload-generator tests
test:
uv run --no-sync pytest tests
cd workload-generator && sbt test

generate-tables:
uv run dat generate-tables

generate-schemas:
uv run dat generate-schemas
generate-workloads output="/tmp/dat-workloads":
cd workload-generator && WORKLOAD_OUTPUT_DIR={{output}} sbt "testOnly io.delta.workload.tables.*"
35 changes: 0 additions & 35 deletions pyproject.toml

This file was deleted.

Empty file removed src/dat/__init__.py
Empty file.
Loading
Loading