Skip to content

RELEASE_PROCESS.md

Latest commit

 

History

History
218 lines (156 loc) · 7.19 KB

RELEASE_PROCESS.md

File metadata and controls

218 lines (156 loc) · 7.19 KB

AppWorld Release Process

This document describes the procedure for releasing new versions of AppWorld data, pypi library and Docker image.

Data

Let's say you want to migrate from 0.1.0 to 0.2.0. The process is as follows:

1. Update scripts/migrate_data.py as needed.

2. Make sure you have saved APPWORLD_PARTIAL_DATA_KEY and APPWORLD_FULL_DATA_KEY env variables.

These are to be obtained from the AWS S3 bucket.

3. Bump the version in code.

Bump the DATA_VERSION version in src/appworld/common/constants.py and update COMPATIBLE_DATA_VERSIONS, if needed appropriate. Note that the version bump has to be made before the release_data script is run because it updates data and DB versions in saved tasks as per this constant.

4. Run the migration script to download the 0.1.0 data in the full mode in data/ directory, apply in-place migration operations to it.

rm -rf data && python scripts/migrate_data.py "0.1.0->0.2.0"

5. Run the release script to process the data into sub-releases and save it in .release.

rm -rf .release && python scripts/release_data.py --ignore-file-diffs  # If there are diffs reported review and make sure they are fine.

6. Check the bundles have only what is intended.

python scripts/show_bundle.py .release/data-0.2.0.bundle
python scripts/show_bundle.py .release/data-0.2.0-with-full-test-ground-truth-${APPWORLD_PARTIAL_DATA_KEY}.bundle
python scripts/show_bundle.py .release/data-0.2.0-with-full-test-ground-truth-${APPWORLD_FULL_DATA_KEY}.bundle

7. Verify that all tasks are passing.

# TODO: Do this for all released versions of appworld pypi package and the latest version on GitHub:
rm -rf .release/data && cp -r .release/data-0.2.0-with-full-test-ground-truth-${APPWORLD_FULL_DATA_KEY} .release/data
appworld verify tasks --include-test-tasks --root .release --num-processes 20
find .release/data -type d -name "__pycache__" -exec rm -r {} +  # To remove pycache that'd build with evaluation runs.

8. Download a prior release in full mode.

rm -rf .previous-release && appworld download data --version 0.1.0 --mode full --root .previous-release

9. Check diffs between two directories to see if it matches expectation.

mkdir -p .tmp/diffs
# Check file-wise diffs
diff -qr .previous-release/data .release/data > .tmp/diffs/overall.diff
# Check sqlite DB diffs for base DBs, if relevant (might need 'brew install sqldiff')
for app in supervisor api_docs admin amazon file_system gmail phone simple_note splitwise spotify todoist venmo; do
  sqldiff .previous-release/data/base_dbs/${app}.db .release/data/base_dbs/${app}.db > .tmp/diffs/db-$app.log
done
# Review: .tmp/diffs

10. The following should pass, or raise an imcompatibility error message as latest COMPATIBLE_DATA_VERSIONS.

appworld verify tasks --root .previous-release

11. Do the actual release on AWS s3.

rm -rf .release && python scripts/release_data.py --ignore-file-diffs --upload

12. As a yet another sanity check, verify everything works with the new release.

# public release
appworld download data && appworld verify tasks --num-processes 20
# private release - 1
appworld download data --mode partial && appworld verify tasks --num-processes 20
# private release - 2
appworld download data --mode full && appworld verify tasks --include-test-tasks --num-processes 20

Experiment Outputs

  • To be added

PyPi

1. Make sure type hints and docstrings, and necessary bundles are updated.

python generate/code/type_hints.py
python generate/code/response_docstrings.py all
python scripts/pack_source_bundles.py

2. Decide the release versions.

Decide the version to release for both appworld and appworld-agents packages.

uv version  # for appworld
uv version --directory experiments  # for appworld-agents

Typically, it'd involve removing the .dev0 suffix, but might involve changing it completely, if it's a bigger release.

3. Publish the packages on test pypi

Before releasing in production pypi, release on test pypi first with release candidate suffixes. E.g., to prepare for 0.2.0 release, release 0.2.0rc0, on test pypi. If something goes wrong, try out 0.2.0rc1 and so on.

# Set RC versions in variables
APPWORLD_VERSION_RC=TODO
APPWORLD_AGENTS_VERSION_RC=TODO

# Update versions in code
uv version $APPWORLD_VERSION_RC
uv version $APPWORLD_AGENTS_VERSION_RC --directory experiments

# Clean old artifacts (optional but recommended)
rm -rf dist experiments/dist

# Build
uv build  # outputs to dist/
uv build --directory experiments # outputs to experiments/dist/

# Spot check (especially that no unexpected files are added)
unzip -l dist/appworld_agents-$APPWORLD_VERSION_RC-py3-none-any.whl
unzip -l experiments/dist/appworld_agents-$APPWORLD_AGENTS_VERSION_RC-py3-none-any.whl

# Publish to TestPyPI
uv publish --publish-url https://test.pypi.org/legacy/
uv publish --directory experiments --publish-url https://test.pypi.org/legacy/

# Verify installs in a fresh env
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple appworld==$APPWORLD_VERSION_RC
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple appworld-agents==$APPWORLD_AGENTS_VERSION_RC
python scripts/verify_package_versions.py
appworld download data
appworld verify tests
appworld verify tasks

4. Publish the packages on production pypi

NOTE: Make sure versions are updated everywhere. PyPi does not allow deleting released packages.

# Set production versions in variables
APPWORLD_VERSION=TODO
APPWORLD_AGENTS_VERSION=TODO

# Update versions in code
uv version $APPWORLD_VERSION
uv version $APPWORLD_AGENTS_VERSION --directory experiments

# Clean old artifacts (optional but recommended)
rm -rf dist experiments/dist

# Build
uv build  # outputs to dist/
uv build --directory experiments # outputs to experiments/dist/

# Publish to PyPI
uv publish
uv publish --directory experiments

# Verify installs in a fresh env
pip install $APPWORLD_VERSION
pip install appworld-agents==$APPWORLD_AGENTS_VERSION
python scripts/verify_package_versions.py
appworld download data
appworld verify tests
appworld verify tasks

5. Bump the dev versions

Whatever versions you released, increment them and add .dev0 prefix.

# Set next dev versions in variables
APPWORLD_VERSION_NEXT_DEV=TODO
APPWORLD_AGENTS_VERSION_NEXT_DEV=TODO

# Update versions in code
uv version $APPWORLD_VERSION_NEXT_DEV
uv version $APPWORLD_AGENTS_VERSION_NEXT_DEV --directory experiments

GitHub

Go to the Release Form.

  • In Select tag, write the current latest tag and hit "Create vX.X.X on publish".
  • Set Target branch to main.
  • Set release title to "Release vX.X.X.
  • Optionally, auto-generate release notes.
  • Preferrable, write up a proper CHANGELOG.md at the top.
  • Set it as the latest release, and publish.

Docker

# Build and push the latest version name release (remember the v prefix)
python scripts/build_image.py --tag vX.X.X --no-cache --push

# Build and push with the latest tag
python scripts/build_image.py --tag latest --no-cache --push