diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 28d3ebd7..368cf891 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -85,8 +85,9 @@ wads/ 1. **`populate my-project`** reads the template at `wads/data/pyproject_toml_tpl.toml` 2. Loads it as TOML, merges user-provided values (name, description, author, license, etc.) 3. Writes the resulting `pyproject.toml` with Hatchling build system -4. Copies `wads/data/github_ci_uv_stub.yml` → `.github/workflows/ci.yml` (5-line stub - that calls the reusable workflow in `i2mint/wads/.github/workflows/uv-ci.yml@master`). +4. Renders `wads/data/github_ci_uv_stub.yml` → `.github/workflows/ci.yml` (5-line stub + that calls the reusable workflow in `i2mint/wads/.github/workflows/uv-ci.yml@master`), + switching its secrets block to the named transport (see "Secrets" below). For repos that need to customize CI beyond `[tool.wads.ci.*]`, drop the stub and copy `wads/data/github_ci_uv.yml` inline instead. 5. Creates README.md, LICENSE, .gitignore, .gitattributes, .editorconfig, package dir @@ -101,7 +102,7 @@ consumer on their next CI run — no per-repo edit, no `wads-migrate` sweep. | Tradeoff | Pin strategy | |---|---| | Float with wads (default) | `@master` — convenient; bad wads merge breaks CI everywhere on next run, but never reaches PyPI (publish is gated on workflow success) | -| Freeze | `@0.2.15` (or any later tag; no `v` prefix) — set via `wads-migrate ci-to-stub --pin @0.2.15`; the repo only picks up wads updates when re-pinned. A JSON-transport stub needs a tag whose uv-ci.yml declares `WADS_CI_SECRETS_JSON` (releases after 0.2.14); older pins need `--transport named` | +| Freeze | `@0.2.15` (or any later tag; no `v` prefix) — set via `wads-migrate ci-to-stub --pin @0.2.15`; the repo only picks up wads updates when re-pinned. A named-transport stub (the default) works with any tag; a JSON-transport stub needs a tag whose uv-ci.yml declares `WADS_CI_SECRETS_JSON` (releases after 0.2.14) | The "CI failure ≠ broken release" property is what makes `@master` safe by default: a botched wads update blocks publication of all downstream packages @@ -119,19 +120,22 @@ A reusable workflow's secret *interface* (`on.workflow_call.secrets`) must be static YAML and `secrets: inherit` is unreliable cross-owner, so secrets are handled in two decoupled layers: -1. **Transport** — the modern stub passes ONE statically-declared secret, - `WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}` — the caller's whole - secrets context, double-encoded so the value is single-line (a multiline - secret is masked per line, and the pretty-printed `{`/`}` lines would - become global masks mangling every brace in the log). Any secret name a - repo has reaches the workflow — nothing to enumerate, nothing to fall - outside of. The 62-name superset (`wads.ci_secrets.DEFAULT_CI_SECRETS`) is - still declared in `uv-ci.yml` but **frozen**: it exists only so old - named-transport stubs keep working (a test in `test_ci_secrets.py` pins the - YAML to `WORKFLOW_CALL_SECRETS` = JSON secret + superset). Named transport - remains available for minimal-secret-surface repos via - `wads-migrate ci-to-stub --transport named`, which warns loudly on - out-of-superset names (they make the workflow unstartable — issue #63). +1. **Transport** — every stub wads *writes* (`populate`, `wads-migrate + ci-to-stub` on an inline workflow) passes secrets **by name**: + `PYPI_PASSWORD` plus the backing secret of each `[tool.wads.ci.env]` var + (`wads.ci_secrets.stub_with_named_transport`). Names must be in the frozen + 62-name superset (`wads.ci_secrets.DEFAULT_CI_SECRETS`, declared in + `uv-ci.yml`; a test in `test_ci_secrets.py` pins the YAML to + `WORKFLOW_CALL_SECRETS` = JSON secret + superset), or the workflow is + unstartable (issue #63), so the CLIs warn loudly on out-of-superset names. + The **opt-in** JSON transport (`--transport json`) passes ONE secret, + `WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}` — the whole secrets + context, double-encoded so the value is single-line (a multiline secret is + masked per line). Any name works, but GitHub's malicious-workflow scanner + holds its runs on NEW repos (`action_required`, zero jobs, no log; issues + #74, #88), which is why it stopped being the default. Re-rendering an + existing stub (`migrate_ci_to_stub(path)`, `ci-to-stub`, `ci-on-demand`, + `fleet-stub`) keeps the transport it already has, with one exception: when no `--transport` is given and a named stub would pass a name outside the superset (so it could never start), the JSON transport is used instead, with a note on stderr. 2. **Env-assignment** — *which* values become job env vars (and which are required) is driven entirely by `[tool.wads.ci.env]` (`required_envvars`, `test_envvars`, `extra_envvars`, `defaults`, and `secret_aliases` for @@ -148,7 +152,7 @@ handled in two decoupled layers: transport feeds it. To use a secret: `wads-secrets add VAR_NAME [SECRET_NAME]` updates pyproject -(and, on legacy named stubs, the stub's pass-through) and can `gh secret set` +(and, on named-transport stubs, the stub's pass-through) and can `gh secret set` the value. For non-sensitive values use `wads-secrets add NAME --variable` (repo variable; no transport, no masking) or put a literal in `[tool.wads.ci.env].defaults`. @@ -298,15 +302,16 @@ wads-test-analyze results.xml ## Testing ```bash -pytest wads/tests/ +uv pip install -e ".[create,docs,skills,test]" # the suite needs the create extra +python -m pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' --ignore=examples --ignore=scrap ``` -Tests are in `wads/tests/` (not the top-level `tests/` directory). +That is CI's exact command. With no path, `testpaths = ["wads"]` collects the tests in `wads/tests/` (not a top-level `tests/`) plus every package doctest; the root `conftest.py` keeps the `wads/data` templates out. The `-o` flag replaces the repo's `NORMALIZE_WHITESPACE`, so a wrapped doctest output needs an inline `# doctest: +NORMALIZE_WHITESPACE`. CI tests 3.10 and 3.12; run 3.11 too before merging. The maintainer skill `skills/wads-dev-workflow` (linked from `.claude/skills/`) has the full checklist: goldens, the push-back test harness, what must change together, and the dependent (`i2mint/isee`) gate. ## Common Pitfalls - The `[tool.wads.ci]` section is **not** standard TOML metadata - it's wads-specific -- `testpaths` in the template defaults to `["tests"]` but wads itself uses `["wads/tests"]` +- `testpaths` in the template defaults to `["tests"]`; wads itself uses `["wads"]` so CI (which runs pytest with no path) also collects the package doctests (#56). The root `conftest.py` keeps `wads/data` templates out of collection - The CI workflow uses `i2mint/wads/actions/*@master` - these must be on GitHub - System deps in `[tool.wads.ops.*]` only run in CI, not locally - Version bumping happens automatically in the publish job on main/master diff --git a/.claude/skills/setup-py-project b/.claude/skills/setup-py-project new file mode 120000 index 00000000..447ff8fe --- /dev/null +++ b/.claude/skills/setup-py-project @@ -0,0 +1 @@ +../../wads/data/skills/setup-py-project \ No newline at end of file diff --git a/.claude/skills/wads-dev-workflow b/.claude/skills/wads-dev-workflow new file mode 120000 index 00000000..bc3a911e --- /dev/null +++ b/.claude/skills/wads-dev-workflow @@ -0,0 +1 @@ +../../skills/wads-dev-workflow \ No newline at end of file diff --git a/.claude/skills/wads-migrate b/.claude/skills/wads-migrate new file mode 120000 index 00000000..38d8b750 --- /dev/null +++ b/.claude/skills/wads-migrate @@ -0,0 +1 @@ +../../wads/data/skills/wads-migrate \ No newline at end of file diff --git a/README.md b/README.md index 3958b6c2..1c588d3a 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,56 @@ # wads -Modern Python project packaging and CI/CD tools for developers who want to focus on code, not configuration. +wads creates, configures and publishes Python packages: a new project gets a Hatchling `pyproject.toml` and a five-line GitHub Actions stub that calls one shared reusable workflow, and every CI setting lives in `[tool.wads.ci]`. It also migrates legacy `setup.cfg` projects, wires CI secrets, and diagnoses CI failures. + +*Still typing code with your own fingers? Skip to [For carbon-based contributors](#for-carbon-based-contributors).* [![PyPI version](https://img.shields.io/pypi/v/wads.svg)](https://pypi.org/project/wads/) [![Python versions](https://img.shields.io/pypi/pyversions/wads.svg)](https://pypi.org/project/wads/) +## For AI agents + +**Skills.** Twelve agent skills ship inside the package, in `wads/data/skills/`: `setup-py-project`, `wads-migrate`, `wads-repo-doctor`, `wads-ci-health`, `wads-changelog`, `wads-docs-coverage`, `wads-docstring-render`, `wads-import-time`, `wads-pypi-polish`, `wads-skillify`, `wads-test-coverage` and `wads-type-coverage`. Enable them either way: + +```bash +pip install "wads[create]" && wads-install-skills # symlinks them into ~/.claude/skills/ +gh skill install i2mint/wads wads-repo-doctor # one skill, from GitHub +``` + +`wads-install-skills --list` prints the available names. In a clone of this repo, `.claude/skills/` links every skill for Claude Code, plus the maintainer skill `wads-dev-workflow` (in `skills/`), which covers changing wads itself. + +**Project instructions.** [`.claude/CLAUDE.md`](.claude/CLAUDE.md) explains the architecture (pyproject as the single source of truth, the reusable workflow, the two-layer secrets model), the config sections, and the conventions. + +**What an agent can do with wads:** + +- Scaffold a package with `populate`, or a whole repo with the `setup-py-project` skill. +- Move a legacy repo to `pyproject.toml` and the CI stub with `wads-migrate`. +- Declare CI secrets and env vars with `wads-secrets add NAME`. +- Read or render a repo's CI configuration from Python, as below. +- Run the CI plan locally with `wads ci-local`, and audit dependency licences with `wads-licence-check`. +- Diagnose a failed run with `wads-ci-debug owner/repo`. + +A minimal example, runnable with only the light core (`pip install wads`): + +```python +from wads.ci_config import CIConfig +from wads.migration import migrate_ci_to_stub + +config = CIConfig( + { + "project": {"name": "mypkg", "optional-dependencies": {"dev": ["httpx"]}}, + "tool": {"wads": {"ci": {"testing": {"python_versions": ["3.12"]}}}}, + } +) +assert config.python_versions == ["3.12"] +assert config.project_name == "mypkg" +# A dev extra CI would never install (no [tool.wads.ci.install].extras): +assert config.uninstalled_test_extras == {"dev": ["httpx"]} + +stub = migrate_ci_to_stub() # the ci.yml a new repo gets +assert "uses: i2mint/wads/.github/workflows/uv-ci.yml@master" in stub +assert "PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}" in stub +``` + ## What is Wads? Wads helps you: @@ -136,26 +182,9 @@ wads-secrets add TEST_LEVEL --variable # non-sensitive value -> repo variab wads-secrets list # show what's configured ``` -`wads-secrets add` (a) records the variable in `[tool.wads.ci.env]` and (b) -runs `gh secret set` (or `gh variable set` with `--variable`) if `gh` is -installed (value taken from `$VAR_NAME` or `--value`). Under the hood there -are two layers: a **transport** — the stub passes your repo's whole secrets -context to the reusable workflow as one `WADS_CI_SECRETS_JSON` secret, so any -secret name works — and an **env policy** (`[tool.wads.ci.env]` — -`required_envvars` / `test_envvars` / `extra_envvars` / `defaults` / -`secret_aliases`) that decides which values become job env vars. Each declared -name resolves against secrets first, then repository *variables* (the right -home for non-sensitive values); committed constants can go straight into -`[tool.wads.ci.env].defaults`. A `required` name that resolves to nothing -fails the build; an undeclared secret is never written to the environment. -Note the JSON transport hands **every** secret the repo can read — including -org-level ones — to the reusable workflow (which only exports the declared -ones). If you want the workflow to receive *only* the names you list, use -`wads-migrate ci-to-stub --transport named` — that mode is limited to the -frozen superset in `wads.ci_secrets.DEFAULT_CI_SECRETS`, and is also the -right choice for orgs with very large shared secrets (the serialized context -must fit in one secret value). Older stubs pass secrets by name the same way; -regenerate with `wads-migrate ci-to-stub` to switch to the JSON transport. +`wads-secrets add` (a) records the variable in `[tool.wads.ci.env]`, (b) adds its secret to the stub's `secrets:` list, and (c) runs `gh secret set` (or `gh variable set` with `--variable`) if `gh` is installed (value taken from `$VAR_NAME` or `--value`). Under the hood there are two layers: a **transport**, where the stub passes secrets to the reusable workflow by name (`PYPI_PASSWORD` plus each declared one), and an **env policy** (`[tool.wads.ci.env]`: `required_envvars` / `test_envvars` / `extra_envvars` / `defaults` / `secret_aliases`) that decides which values become job env vars. Each declared name resolves against secrets first, then repository *variables* (the right home for non-sensitive values); committed constants can go straight into `[tool.wads.ci.env].defaults`. A `required` name that resolves to nothing fails the build; an undeclared secret is never written to the environment. + +Named secrets must be in the frozen superset in `wads.ci_secrets.DEFAULT_CI_SECRETS`, or GitHub rejects the workflow at parse time; `wads-secrets` and `wads-migrate` warn about such names. The opt-in alternative, `wads-migrate ci-to-stub --transport json`, passes the repo's whole secrets context as one `WADS_CI_SECRETS_JSON` secret, so any name works. It is not the default because GitHub's malicious-workflow scanner holds its runs on new repositories: every run ends `action_required` with zero jobs and no log ([#74](https://github.com/i2mint/wads/issues/74)). Re-rendering an existing stub keeps whichever transport it already uses. ### Declare System Dependencies @@ -563,37 +592,14 @@ note = "On Alpine: apk add unixodbc unixodbc-dev" alternatives = ["iodbc"] ``` -See [docs/SYSTEM_DEPENDENCIES.md](docs/SYSTEM_DEPENDENCIES.md) for comprehensive examples. - -## Claude Code Skills - -Wads ships with [Claude Code](https://docs.anthropic.com/en/docs/claude-code) skills for AI-assisted workflows. Install them globally so they're available in every project: - -```bash -wads-install-skills -``` - -This symlinks skills to `~/.claude/skills/`, so they stay in sync when wads is updated: - -| Command | Description | -|---------|-------------| -| `/setup-py-project` | AI-guided Python project creation: name suggestions, PyPI/GitHub availability checking, repo creation, file population | -| `/wads-migrate` | Migrate projects to modern wads setup (pyproject.toml + uv CI) | - -**Example:** -``` -/setup-py-project "a tool for audio signal processing" -``` - -To list available skills without installing: `wads-install-skills --list` -To update existing skills: `wads-install-skills --force` +See [misc/docs/SYSTEM_DEPENDENCIES.md](misc/docs/SYSTEM_DEPENDENCIES.md) for comprehensive examples. ## Documentation - **[System Dependencies Guide](misc/docs/SYSTEM_DEPENDENCIES.md)** - `[tool.wads.ops.*]` format and examples - **[Migration Guide](misc/docs/MIGRATION.md)** - Migrate from setup.cfg to pyproject.toml - **[Utilities Reference](misc/docs/UTILITIES.md)** - CLI tools (`wads-ci-debug`, `wads-migrate`) -- **[CLAUDE.md](CLAUDE.md)** - AI agent guide for working with this project +- **[.claude/CLAUDE.md](.claude/CLAUDE.md)** - AI agent guide for working with this project ## Troubleshooting @@ -630,21 +636,31 @@ Common issues: - Python version incompatibilities → Check `python_versions` in `[tool.wads.ci.testing]` - Test failures → Review generated fix instructions -## Development +## For carbon-based contributors + +Everything above is for users of wads, human or not. This part is for working on wads itself. -### Running Tests +**Dev setup.** The test suite scaffolds and builds packages, so it needs the `create` extra: ```bash -pytest wads/tests/ +uv venv && . .venv/bin/activate +uv pip install -e ".[create,docs,skills,test]" ``` -### Building Documentation +**Run the tests the way CI does.** CI calls pytest with no path, so `testpaths = ["wads"]` collects both `wads/tests` and every doctest in the package: ```bash -pip install -e ".[docs]" -epythet build +python -m pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' --ignore=examples --ignore=scrap ``` +CI tests Python 3.10 and 3.12; run 3.11 as well before merging, because a dataclass default once broke there and nowhere else ([#100](https://github.com/i2mint/wads/issues/100)). + +**Build the docs** with `pip install -e ".[docs]"` and then `epythet build`. + +**Why it is built this way.** wads is a foundation package. Its reusable workflow and actions run from `@master` in every wads-managed repo, and every merge to `master` publishes a release to PyPI. So changes to `.github/workflows/uv-ci.yml`, `actions/*` or the stub template are fleet-wide changes: they come with tests (the push-back script, for example, is exercised against real throwaway git repos), and the populate output is pinned by golden files. [`.claude/CLAUDE.md`](.claude/CLAUDE.md) has the design rationale, and the `wads-dev-workflow` skill has the maintainer checklist. + +**Contributing and questions.** Open an issue or a pull request at [github.com/i2mint/wads](https://github.com/i2mint/wads/issues). + ## License Apache Software License 2.0 diff --git a/actions/git-commit/action.yml b/actions/git-commit/action.yml index 73bcb15d..741ad221 100644 --- a/actions/git-commit/action.yml +++ b/actions/git-commit/action.yml @@ -166,19 +166,67 @@ runs: done if [ -n "$conflicted" ] && [ "$only_version_files" = 1 ]; then echo "Version file conflict with a concurrent change; keeping upstream content, replaying only the version bump in: $conflicted" + # The version line lives in [project] (pyproject.toml) or + # [metadata] (setup.cfg), or before any table in a bare file; + # another table's `version` key (a tool setting) is not it. + version_section() { + case "$1" in + setup.cfg) echo metadata ;; + *) echo project ;; + esac + } + # Shared awk: the first `version =` line in that section. With + # nv empty it prints the line's X.Y.Z; otherwise it rewrites that + # X.Y.Z to nv and prints the whole file. + version_awk=' + /^[[:space:]]*\[[^]]*\]+[[:space:]]*([#;].*)?\r?$/ { + seen = 1 + insec = ($0 ~ ("^[[:space:]]*\\[[[:space:]]*" sec "[[:space:]]*\\]")) + } + !done && (!seen || insec) && /^[[:space:]]*version[[:space:]]*=/ \ + && match($0, /[0-9]+\.[0-9]+\.[0-9]+/) { + done = 1 + if (nv == "") { print substr($0, RSTART, RLENGTH); exit } + $0 = substr($0, 1, RSTART - 1) nv substr($0, RSTART + RLENGTH) + } + nv != "" { print } + ' + # The project's X.Y.Z in file $1's section, read from stdin. + project_version() { + awk -v sec="$(version_section "$1")" -v nv="" "$version_awk" || true + } for path in $conflicted; do - theirs_content=$(git show ":3:$path" 2>/dev/null || true) - new_version=$(printf '%s\n' "$theirs_content" \ - | grep -m1 -E '^[[:space:]]*version[[:space:]]*=' \ - | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 || true) - git checkout --ours -- "$path" + new_version=$(git show ":3:$path" 2>/dev/null | project_version "$path" || true) + if ! git checkout --ours -- "$path" 2>/dev/null; then + # Modify/delete conflict: upstream removed the file, so there + # is no upstream content to keep (i2mint/wads#98). + git rebase --abort || true + echo "::error::'$path' was deleted or renamed on 'origin/$branch' while this run bumped it to ${new_version:-a new version}. Resolve by hand." + exit 1 + fi if [ -n "$new_version" ]; then case "$path" in - pyproject.toml) - sed -i -E "s/^version = \"[0-9]+\.[0-9]+\.[0-9]+\"/version = \"$new_version\"/" "$path" - ;; - setup.cfg) - sed -i -E "s/^version = [0-9]+\.[0-9]+\.[0-9]+/version = $new_version/" "$path" + pyproject.toml|setup.cfg) + # Rewrite only the X.Y.Z of the project's version line -- + # the same line the version was read from -- keeping the + # file's own spacing and quoting (i2mint/wads#98: the old + # `sed` matched only `version = "X.Y.Z"` verbatim, so any + # other valid spelling was a silent no-op, and it also + # rewrote every other table's `version = "..."` line). + # awk + mv rather than `sed -i`, whose flags differ + # between GNU and BSD sed. + awk -v sec="$(version_section "$path")" -v nv="$new_version" \ + "$version_awk" "$path" > "$path.wads-replay" + mv -f "$path.wads-replay" "$path" + landed=$(project_version "$path" < "$path") + if [ "$landed" != "$new_version" ]; then + # Pushing now would record an older version in git than + # the one this run published (the i2mint/wads#83 desync), + # so stop loudly instead. + git rebase --abort || true + echo "::error::Could not write version $new_version (just published) into '$path' on top of 'origin/$branch' (found '${landed:-no version line}'). Resolve by hand." + exit 1 + fi ;; setup.py) # No fixed version-field syntax to target safely; skip -- @@ -186,6 +234,8 @@ runs: # skipping never loses data (unlike the old --theirs). ;; esac + else + echo "::warning::No X.Y.Z version found in this run's '$path'; kept upstream's version line as is." fi git add -- "$path" done diff --git a/conftest.py b/conftest.py new file mode 100644 index 00000000..6ee46fd3 --- /dev/null +++ b/conftest.py @@ -0,0 +1,8 @@ +"""Repository-level pytest configuration. + +``wads/data`` holds project *templates* (for example ``test_smoke_tpl.py``, +which is not valid Python until rendered), so pytest must never import it, +whatever directory pytest is run from. +""" + +collect_ignore = ["wads/data"] diff --git a/pyproject.toml b/pyproject.toml index 67333018..9e546971 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,8 +8,34 @@ version = "0.2.31" description = "Tools for packaging and publishing to pypi for those who just do not want to deal with it" readme = "README.md" requires-python = ">=3.10" -keywords = ["documentation", "packaging", "publishing"] +license = "Apache-2.0" +keywords = [ + "packaging", + "publishing", + "pypi", + "pyproject", + "hatchling", + "ci", + "github-actions", + "uv", + "project-template", + "documentation", +] authors = [{ name = "Thor Whalen" }] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "Operating System :: OS Independent", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Software Development :: Build Tools", + "Topic :: Software Development :: Code Generators", + "Topic :: System :: Software Distribution", +] # Core (light) dependencies: enough to read config ([tool.wads.ci] / # package.json wads.ci), run the templating engine, and `import wads`. Kept # deliberately small so CI-side use stays cheap. Heavier project-creation / @@ -23,9 +49,6 @@ dependencies = [ "tomli-w>=0.4.0", ] -[project.license] -text = "Apache Software License" - [project.optional-dependencies] # Full project-creation / publishing toolchain (heavier). Install with # `pip install wads[create]` when scaffolding or publishing projects. @@ -46,6 +69,9 @@ all = ["wads[create,docs]"] [project.urls] Homepage = "https://github.com/i2mint/wads" +Repository = "https://github.com/i2mint/wads" +Documentation = "https://i2mint.github.io/wads/" +Issues = "https://github.com/i2mint/wads/issues" [project.scripts] pack = "wads.pack:main" @@ -98,7 +124,11 @@ select = ["D100"] [tool.pytest.ini_options] minversion = "6.0" -testpaths = ["wads/tests"] +# The whole package, not just wads/tests: run-tests-uv calls pytest with no +# path, so testpaths alone decides what CI collects, and the package's own +# doctests must be in it (i2mint/wads#56). wads/data (templates) is left out +# by collect_ignore in the root conftest.py. +testpaths = ["wads"] doctest_optionflags = ["NORMALIZE_WHITESPACE", "ELLIPSIS"] [tool.wads.ci.install] diff --git a/skills/wads-dev-workflow/SKILL.md b/skills/wads-dev-workflow/SKILL.md new file mode 100644 index 00000000..cceb7127 --- /dev/null +++ b/skills/wads-dev-workflow/SKILL.md @@ -0,0 +1,84 @@ +--- +name: wads-dev-workflow +description: >- + How to change the wads package itself (i2mint/wads) safely: install the right + extras, run the test suite exactly the way CI does (package doctests + included, on Python 3.10/3.11/3.12), regenerate the populate goldens after an + intended output change, exercise the git-commit push-back script, and keep + the reusable workflow, actions, stub template and secrets superset in sync. + Use when editing wads source, actions/*, .github/workflows/uv-ci.yml, + wads/data templates or the shipped skills, when a wads test fails, when + "regenerate the goldens", "run wads tests like CI", or "is this wads change + safe to merge" comes up. Every merge to master publishes wads to PyPI and + goes live for every repo whose stub floats on @master. Not for USING wads in + another repo (see wads-migrate, wads-ci-health, setup-py-project). +metadata: + audience: developers +--- + +# Working on wads itself + +wads is a foundation package: its reusable workflow (`.github/workflows/uv-ci.yml`) and composite actions (`actions/*`) run from `@master` in every wads-managed repo, and a merge to `master` publishes to PyPI. Treat any change to a workflow, action, or stub template as a fleet-wide change. + +## Setup + +```bash +uv venv -q .venv && . .venv/bin/activate +uv pip install -e ".[create,docs,skills,test]" pytest pip +``` + +The default `.[test,dev]` is not enough: the suite builds and scaffolds packages, which needs the `create` extra (`requests`, `build`, `ruamel.yaml`, `tomlkit`). + +## Run the tests the way CI does + +CI (`actions/run-tests-uv`) runs pytest with **no path**, so `[tool.pytest.ini_options].testpaths = ["wads"]` decides what is collected: `wads/tests` plus every package doctest. The root `conftest.py` keeps the `wads/data` templates out. + +```bash +python -m pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' --ignore=examples --ignore=scrap -q +``` + +- The `-o doctest_optionflags=...` **replaces** the repo's `NORMALIZE_WHITESPACE`, so a doctest whose expected output wraps across lines needs an inline `# doctest: +NORMALIZE_WHITESPACE`. +- The CI matrix is 3.10 and 3.12, but users run 3.11 too. Before a merge, run the command on all three (`uv venv -p python3.10 …`); 3.11 alone rejects unhashable dataclass defaults, and 3.10's `mock.patch` resolves dotted paths by attribute, which catches `sys.modules` leaks between tests. +- `wads/tests/test_collection_scope.py` pins that package doctests stay collected. + +## Goldens: intended output changes + +`wads/tests/test_populate_characterization.py` compares a default `populate` byte-for-byte with `wads/tests/data/golden/python_lib/`. When you change default output on purpose, regenerate from the real output and review the diff: + +```python +import subprocess, tempfile, shutil +from pathlib import Path +from wads.populate import populate_pkg_dir + +with tempfile.TemporaryDirectory() as d: + pkg = Path(d) / "mypkg"; pkg.mkdir() + for cmd in (["git", "init", "-q"], ["git", "config", "user.email", "t@e.com"], + ["git", "config", "user.name", "t"], + ["git", "remote", "add", "origin", "https://github.com/myorg/mypkg"]): + subprocess.run(cmd, cwd=pkg, check=True) + populate_pkg_dir(str(pkg), description="Test package", root_url="https://github.com/myorg", + author="John Doe", version="1.2.3", verbose=False) + for rel in ("pyproject.toml", ".github/workflows/ci.yml"): + shutil.copy(pkg / rel, Path("wads/tests/data/golden/python_lib") / rel) +``` + +`git diff wads/tests/data/golden/` must show only the change you meant. The `frontend_js` golden is byte-identical by contract; don't regenerate it casually. + +## The push-back script (`actions/git-commit`) + +`wads/tests/test_git_commit_push_retry.py` runs the action's real "Push Changes" bash step (read out of `action.yml`) against throwaway bare repos, including concurrent version-bump races (issues #81, #83, #89, #98). Add a case there for any change to that step. Keep the script portable: no `sed -i` (GNU and BSD differ), POSIX `awk` only (`ubuntu-latest` has mawk). + +## Things that move together + +- **Reusable workflow vs inline template**: `.github/workflows/uv-ci.yml` and `wads/data/github_ci_uv.yml` mirror each other; change both. +- **Secrets interface**: `on.workflow_call.secrets` in `uv-ci.yml` is pinned to `wads.ci_secrets.WORKFLOW_CALL_SECRETS` by `test_ci_secrets.py`; the named superset is frozen. +- **Stub template** `wads/data/github_ci_uv_stub.yml` keeps the JSON-transport region because `stub_with_named_transport` rewrites it; new stubs are rendered with the named transport (issue #74). +- **A new `read-ci-config` output** is empty until a wads release carrying it reaches PyPI (the action `pip install`s wads), while `uv-ci.yml@master` changes go live at merge. Guard new `if:` clauses against an empty output. + +## Before merging + +1. The CI-exact command passes on 3.10, 3.11 and 3.12. +2. `uvx ruff check .` and `uvx ruff format --check .` pass (CI formats on publish, but keep diffs clean). +3. If packaging changed, build from tracked files only and run `twine check` (a stray local venv leaks into a local sdist). +4. Run the dependent `i2mint/isee` tests with this checkout installed editable over it. +5. Never write CI markers (the publish or skip markers) in commit messages or PR text; they are directives. diff --git a/wads/ci_config.py b/wads/ci_config.py index 45866654..f607a8f3 100644 --- a/wads/ci_config.py +++ b/wads/ci_config.py @@ -116,6 +116,62 @@ def install_extras(self) -> str: items = [str(e).strip() for e in extras if str(e).strip()] return ",".join(items) + #: Extra names that conventionally hold test-time tooling (i2mint/wads#59). + TEST_EXTRA_NAMES = ("dev", "test", "tests", "testing", "ci") + #: Packages CI provides without any extra: run-tests-uv installs pytest and + #: pytest-cov (which pulls coverage); ruff comes from the ruff actions. + CI_PROVIDED_PACKAGES = ("pytest", "pytest-cov", "coverage", "ruff") + + @property + def uninstalled_test_extras(self) -> dict[str, list[str]]: + """Test-looking extras CI will not install, mapped to what they would add. + + Non-empty only when ``[tool.wads.ci.install].extras`` is absent: CI then + installs core dependencies only, so an extra named like + :attr:`TEST_EXTRA_NAMES` whose packages go beyond + :attr:`CI_PROVIDED_PACKAGES` is silently missing from the test job + (i2mint/wads#59). Any explicit ``extras`` value, including ``""``, is + a decision and silences this. + + >>> config = CIConfig({'project': {'name': 'p', 'optional-dependencies': { + ... 'dev': ['pytest', 'httpx>=0.27'], 'docs': ['sphinx']}}}) + >>> config.uninstalled_test_extras + {'dev': ['httpx']} + """ + if "extras" in self.ci_config.get("install", {}): + return {} + from packaging.requirements import Requirement + from packaging.utils import canonicalize_name + + own_name = canonicalize_name(self.project_name or "") + provided = {canonicalize_name(n) for n in self.CI_PROVIDED_PACKAGES} + optional = (self.data.get("project") or {}).get("optional-dependencies") + if not isinstance(optional, dict): + return {} + missing = {} + for extra, requirements in optional.items(): + if not isinstance(extra, str) or extra.lower() not in self.TEST_EXTRA_NAMES: + continue + if not isinstance(requirements, list): + continue + names = [] + for requirement in requirements: + if not isinstance(requirement, str): + continue + try: + parsed = Requirement(requirement) + if parsed.marker is not None and not parsed.marker.evaluate( + {"extra": extra} + ): + continue # does not apply to this interpreter/platform + except Exception: # invalid requirement or marker: not ours to judge + continue + if canonicalize_name(parsed.name) not in provided | {own_name}: + names.append(parsed.name) + if names: + missing[extra] = names + return missing + # ⚙️ EXECUTION FLOW AND COMMANDS @property def commands_pre_test(self) -> list[str]: diff --git a/wads/ci_secrets.py b/wads/ci_secrets.py index 28522836..e93275d0 100644 --- a/wads/ci_secrets.py +++ b/wads/ci_secrets.py @@ -4,8 +4,23 @@ CI secrets: what the reusable workflow (``uv-ci.yml``) declares in ``on.workflow_call.secrets`` and what the caller stub passes. -Transport: one JSON secret (the modern default) ------------------------------------------------ +Transport: named by default, one JSON secret on request +------------------------------------------------------- +Every stub wads WRITES (``populate``, ``wads-migrate ci-to-stub`` on an inline +workflow) passes its secrets by name: ``PYPI_PASSWORD`` plus the backing secret +of each env var declared in ``[tool.wads.ci.env]``. The JSON transport below +remains available with ``wads-migrate ci-to-stub --transport json``, and an +existing stub keeps whichever transport it has when re-rendered. + +Why named is the default (issues #74, #88): serialising the whole ``secrets`` +context into a workflow in another repository is structurally what a +secret-exfiltration workflow looks like, and GitHub's malicious-workflow +scanner holds such runs on NEW repositories -- ``action_required``, zero jobs, +no log, and the REST approve endpoint refuses them. It was reproduced on four +new repositories; switching to the named transport made the next push run. + +The JSON transport +^^^^^^^^^^^^^^^^^^ A GitHub *reusable* workflow's secret interface (``on.workflow_call.secrets``) must be **static YAML** — it is parsed before any job runs and cannot be parametrized from ``pyproject.toml``. ``secrets: inherit`` is documented to @@ -41,23 +56,22 @@ consumer's ``pyproject.toml`` (see :mod:`wads.ci_config` and the ``export-ci-env`` action). Nothing is exported unless declared there. -The named superset (legacy transport, kept for back-compat) ------------------------------------------------------------ -Before the JSON transport, the workflow declared a generous *superset* of -optional secret names (:data:`DEFAULT_CI_SECRETS`) and each repo's stub passed -a named subset. That design failed whenever a repo needed a name outside the -superset — GitHub rejects an undeclared secret at parse time with an opaque -``startup_failure`` (issue #63). +The named superset (the named transport's universe) +---------------------------------------------------- +The workflow also declares a generous *superset* of optional secret names +(:data:`DEFAULT_CI_SECRETS`), and a named-transport stub passes a subset of +it. A name outside the superset makes GitHub reject the workflow at parse time +with an opaque ``startup_failure`` (issue #63), so ``wads-migrate`` and +``wads-secrets`` warn loudly about such names; a repo that needs one can use a +repository variable (for non-sensitive values) or opt into the JSON transport. -The superset is still declared by ``uv-ci.yml`` so that already-deployed -named-transport stubs keep working, but it is **frozen**: new names should not -be added — a repo that needs a new name should switch to the JSON transport -stub (``wads-migrate ci-to-stub``), which transports everything. +The superset is **frozen**: it is pinned to the YAML by a test, and changing +the reusable workflow's secret interface affects every consumer. So there are two layers: -* **Transport** — the JSON secret (modern) or the frozen superset (legacy), - rendered into static YAML. Plumbing. +* **Transport** — named (default) or the JSON secret (opt-in), rendered into + static YAML. Plumbing. * **Env-assignment** — pyproject-driven, exact, per-repo. The thing users tune. Keeping the names here (Python) and *rendering* them into the YAML (with a @@ -66,6 +80,7 @@ """ import re +import sys as _sys # The single statically-declared secret through which a stub transports the # caller's whole `secrets` context (double-encoded JSON; see module docstring). @@ -325,7 +340,7 @@ def render_workflow_call_secrets( def render_stub_secrets_passthrough( names=DEFAULT_CI_SECRETS, *, indent: int = 6 ) -> str: - """Render a caller stub's *named* ``secrets:`` pass-through block (legacy). + """Render a caller stub's *named* ``secrets:`` pass-through block (default). >>> print(render_stub_secrets_passthrough(["PYPI_PASSWORD", "NPM_TOKEN"])) PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }} @@ -336,10 +351,92 @@ def render_stub_secrets_passthrough( def render_stub_json_transport(*, indent: int = 6) -> str: - """Render the caller stub's JSON-transport ``secrets:`` line (the default). + """Render the caller stub's JSON-transport ``secrets:`` line (opt-in). >>> print(render_stub_json_transport()) WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }} """ pad = " " * indent return f"{pad}{JSON_TRANSPORT_SECRET}: {JSON_TRANSPORT_EXPRESSION}" + + +# The comment + `secrets:` block a named-transport stub carries in place of the +# template's JSON-transport region. +NAMED_TRANSPORT_COMMENT = """\ + # Transport (NAMED, the default): passes only the secrets listed below -- + # PYPI_PASSWORD plus the backing secret of each env var declared in + # [tool.wads.ci.env] (`wads-secrets add VAR_NAME` updates both). Every + # name must be in the frozen wads superset (wads/ci_secrets.py) or GitHub + # rejects the workflow at parse time. + # + # The opt-in JSON transport (`wads-migrate ci-to-stub --transport json`) + # passes every secret without a list, but GitHub's malicious-workflow + # scanner holds runs that use it on new repositories: `action_required`, + # zero jobs, no log (i2mint/wads#74). + # + # *Which* of these become job env vars -- and which are required -- is + # driven by [tool.wads.ci.env] in pyproject.toml. Non-sensitive values + # don't need a secret: use [tool.wads.ci.env].defaults or a repository + # *variable* (`gh variable set NAME`). + secrets: +""" + + +def stub_with_named_transport(stub: str, names) -> str: + r"""Swap a stub's JSON-transport region for a named ``secrets:`` pass-through. + + The region runs from the ``# Transport:`` comment down to the JSON line, so + the result never describes a transport it does not use. + + >>> stub = ( + ... "jobs:\n ci:\n uses: x\n # Transport: whole context.\n" + ... " secrets:\n" + render_stub_json_transport() + "\n" + ... ) + >>> named = stub_with_named_transport(stub, ["PYPI_PASSWORD"]) + >>> named.endswith(" PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}\n") + True + >>> "toJSON" in named + False + """ + json_region = re.compile( + r"^ # Transport:.*?" + re.escape(render_stub_json_transport()) + r"\n", + re.DOTALL | re.MULTILINE, + ) + named_region = ( + NAMED_TRANSPORT_COMMENT + render_stub_secrets_passthrough(names) + "\n" + ) + new_stub, n_replaced = json_region.subn(lambda _: named_region, stub) + if n_replaced != 1: + raise ValueError( + "stub template changed shape: could not locate the JSON " + "transport region to convert to named transport" + ) + return new_stub + + +def warn_names_outside_superset(names) -> list: + """Warn loudly for names a named-transport stub cannot legally pass. + + A caller may only pass secrets the reusable workflow declares. With + ``transport="named"`` that universe is the frozen superset in + :data:`wads.ci_secrets.DEFAULT_CI_SECRETS`; a stub naming anything outside + it produces a workflow GitHub rejects at parse time — zero jobs, an opaque + ``startup_failure`` (issue #63). Returns the offending names. + """ + outside = [n for n in names if n not in DEFAULT_CI_SECRETS] + for name in outside: + print( + f"warning: {name!r} is not in the wads secrets superset, so a stub " + f"passing it by name CANNOT START (GitHub rejects the workflow at " + f"parse time with `startup_failure`). Either:\n" + f" - if the value is not actually sensitive, store it as a " + f"repository VARIABLE (`gh variable set {name}`) — declared env " + f"vars fall back to repo variables automatically; or\n" + f" - opt into the JSON transport (`wads-migrate ci-to-stub " + f"--transport json`), which passes every secret (but GitHub may " + f"hold its runs on a new repo, i2mint/wads#74); or\n" + f" - keep the inline workflow (`wads-migrate ci-to-uv`, don't " + f"stub-ify).", + file=_sys.stderr, + ) + return outside diff --git a/wads/ci_trigger.py b/wads/ci_trigger.py index e7b1c87a..2a68e696 100644 --- a/wads/ci_trigger.py +++ b/wads/ci_trigger.py @@ -336,15 +336,16 @@ def _assign(table, key, value): def stub_shape(ci_text: Optional[str]) -> dict: """The ``pin`` and secrets ``transport`` of a stub, which a re-render must keep. - Defaults (``@master``, ``json``) for anything that is not a stub. + Defaults (``@master``, ``named``) for anything that is not a stub: a NEW + stub gets the named transport (i2mint/wads#74), an existing one keeps its own. >>> stub_shape("uses: i2mint/wads/.github/workflows/uv-ci.yml@0.2.30") {'pin': '@0.2.30', 'transport': 'named'} >>> stub_shape(None) - {'pin': '@master', 'transport': 'json'} + {'pin': '@master', 'transport': 'named'} """ if classify_ci_workflow(ci_text) != "stub": - return {"pin": "@master", "transport": "json"} + return {"pin": "@master", "transport": "named"} from wads.ci_secrets import render_stub_json_transport match = _PIN_RE.search(ci_text) @@ -552,7 +553,8 @@ def flip_to_on_demand( shadow_ci.parent.mkdir(parents=True, exist_ok=True) shadow_ci.write_text(old_ci) shape = stub_shape(old_ci) - new_ci = migrate_ci_to_stub(str(shadow_ci), **shape) + # transport=None: keep an existing stub's, pick one for an inline workflow. + new_ci = migrate_ci_to_stub(str(shadow_ci), pin=shape["pin"]) if shape["pin"] != "@master": result.notes.append( f"the stub stays pinned to {shape['pin']}. Its pre-filter works on " diff --git a/wads/data/github_ci_uv_stub.yml b/wads/data/github_ci_uv_stub.yml index a09825a3..fdb48786 100644 --- a/wads/data/github_ci_uv_stub.yml +++ b/wads/data/github_ci_uv_stub.yml @@ -7,9 +7,9 @@ # Pinning: `@master` floats with wads. If you need version stability for # a release-sensitive repo, change `@master` to a wads tag (e.g. `@0.2.15`; # tags have no `v` prefix). A stub whose `secrets:` block passes the JSON -# transport (the default below) needs a tag from a release after 0.2.14 — -# older tags don't declare that secret and GitHub then rejects the -# workflow at parse time. +# transport (`WADS_CI_SECRETS_JSON`) needs a tag from a release after +# 0.2.14 — older tags don't declare that secret and GitHub then rejects +# the workflow at parse time. The named transport works with any tag. # CI failure does not block a published release — it blocks the publish # step itself — so floating master is generally safe. # @@ -41,8 +41,10 @@ jobs: # propagate secrets, so it cannot replace this.) # # Note this hands EVERY secret this repo can read — including org-level - # ones — to the called workflow. For a minimal secret surface (only the - # names you list), regenerate with + # ones — to the called workflow, and GitHub's malicious-workflow scanner + # holds runs that do this on NEW repositories (`action_required`, zero + # jobs, no log; i2mint/wads#74). That is why generated stubs default to + # the named transport; switch with # wads-migrate ci-to-stub --transport named # # *Which* of these become job env vars — and which are required — is diff --git a/wads/data/skills/wads-migrate/SKILL.md b/wads/data/skills/wads-migrate/SKILL.md index 1dc6f450..5ada63e9 100644 --- a/wads/data/skills/wads-migrate/SKILL.md +++ b/wads/data/skills/wads-migrate/SKILL.md @@ -30,9 +30,9 @@ as an escape valve for repos that need to customize CI beyond `[tool.wads.ci.*]` | Con | Mitigation | |---|---| | Bad wads merge breaks CI everywhere on next run | Wads's own CI runs the reusable workflow first — canary catches obvious breaks. **Crucially: broken CI ≠ broken release.** Publish is gated on workflow success, so a bad wads change blocks publication for downstream consumers until wads is fixed, but never ships a broken artifact. This is what makes floating `@master` safe by default. | -| Floating `@master` means consumers can't pin a known-good wads state | `wads-migrate ci-to-stub --pin @0.2.15` writes the stub with a tag pin instead of `@master` (i2mint tags are bare versions — **no `v` prefix**; `@v0.2.15` would reference a nonexistent ref). A default (JSON-transport) stub needs a tag from a release **after 0.2.14** — older tags don't declare `WADS_CI_SECRETS_JSON` and the workflow is rejected at parse time (the CLI warns loudly if you try). The pinned repo only picks up wads updates when explicitly re-pinned. Use for release-sensitive repos. | -| A secret your tests need isn't reaching CI | Run `wads-secrets add VAR_NAME` (see "Secrets" below). It declares the var in `[tool.wads.ci.env]`; on the default JSON transport nothing else is needed (every repo secret is passed automatically). Only a stub on the opt-in *named* transport needs a pass-through line, which the CLI adds — refusing names outside the wads superset (`wads.ci_secrets.DEFAULT_CI_SECRETS`), since passing one would make the workflow fail to start. | -| Secrets must reach a reusable workflow owned by a different account | The stub passes secrets **explicitly** (NOT `secrets: inherit`, which is unreliable cross-owner). The default is the JSON transport — the whole `secrets` context serialized into the one declared secret `WADS_CI_SECRETS_JSON`. A named-transport stub instead lists each name, generated from `[tool.wads.ci.env]` at migrate time and extended by `wads-secrets add`. | +| Floating `@master` means consumers can't pin a known-good wads state | `wads-migrate ci-to-stub --pin @0.2.15` writes the stub with a tag pin instead of `@master` (i2mint tags are bare versions — **no `v` prefix**; `@v0.2.15` would reference a nonexistent ref). A default (named-transport) stub works with any tag; an opt-in JSON-transport stub needs a tag from a release **after 0.2.14** — older tags don't declare `WADS_CI_SECRETS_JSON` and the workflow is rejected at parse time (the CLI warns loudly if you try). The pinned repo only picks up wads updates when explicitly re-pinned. Use for release-sensitive repos. | +| A secret your tests need isn't reaching CI | Run `wads-secrets add VAR_NAME` (see "Secrets" below). It declares the var in `[tool.wads.ci.env]` and, on the default *named* transport, adds the stub's pass-through line — refusing names outside the wads superset (`wads.ci_secrets.DEFAULT_CI_SECRETS`), since passing one would make the workflow fail to start. A stub on the opt-in JSON transport needs no stub edit (every repo secret is passed automatically). | +| Secrets must reach a reusable workflow owned by a different account | The stub passes secrets **explicitly** (NOT `secrets: inherit`, which is unreliable cross-owner). The default is the *named* transport: the stub lists each name, generated from `[tool.wads.ci.env]` at migrate time and extended by `wads-secrets add`. The opt-in JSON transport (`--transport json`) serializes the whole `secrets` context into the one declared secret `WADS_CI_SECRETS_JSON`, but GitHub's malicious-workflow scanner holds its runs on new repos (`action_required`, zero jobs, no log; wads#74). | ## Detecting Current Format @@ -226,8 +226,8 @@ writes it into the job environment, (b) adds the pass-through line to the stub's `secrets:` block so the secret is transported to the reusable workflow, and (c) `gh secret set`s the value if `gh` is installed (value from `$VAR_NAME` or `--value`). See the **Secrets** section below for the full model. Superset -limits only apply to stubs on the opt-in *named* transport (the CLI refuses an -edit that would make such a stub fail to start); the default JSON transport +limits apply to stubs on the default *named* transport (the CLI refuses an +edit that would make such a stub fail to start); the opt-in JSON transport accepts any secret name. Do NOT hand-edit job-level `env:` blocks in ci.yml. Declare via @@ -251,15 +251,18 @@ the repo's CI shape. Secrets reach a stub repo's CI through two coordinated layers: 1. **Transport** — the stub's `secrets:` block *passes* secrets to the - reusable workflow. The default (since the wads#64 fix) is the **JSON - transport**: the whole `secrets` context serialized into the one declared - secret `WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}` — any secret - name works, there is no fixed list to fall outside of. The opt-in - alternative (`wads-migrate ci-to-stub --transport named`) passes each - secret by name for a minimal surface; named transport is limited to the - superset declared in the wads-side `uv-ci.yml` (`on.workflow_call.secrets`, - generated from `wads.ci_secrets.DEFAULT_CI_SECRETS`) — a stub naming - anything outside it fails at parse time (issue #63). Either way, explicit + reusable workflow. The default is the **named transport**: each secret by + name (`PYPI_PASSWORD` plus the `[tool.wads.ci.env]`-declared ones), limited + to the superset declared in the wads-side `uv-ci.yml` + (`on.workflow_call.secrets`, generated from + `wads.ci_secrets.DEFAULT_CI_SECRETS`) — a stub naming anything outside it + fails at parse time (issue #63). The opt-in **JSON transport** + (`wads-migrate ci-to-stub --transport json`) serializes the whole `secrets` + context into the one declared secret + `WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}`, so any name works, + but GitHub's malicious-workflow scanner holds its runs on NEW repos: every + run ends `action_required` with zero jobs and no log (wads#74, #88). Re- + rendering an existing stub keeps its transport. Either way, explicit pass-through is used, NOT `secrets: inherit` (unreliable across GitHub accounts). 2. **Env-assignment** — `[tool.wads.ci.env]` (`required_envvars`, @@ -272,10 +275,11 @@ Secrets reach a stub repo's CI through two coordinated layers: **`wads-secrets add VAR_NAME [SECRET_NAME]`** does both layers (+ `gh secret set`) in one step — the recommended way. `wads-secrets list` shows what's configured; `wads-secrets superset` prints the names a *named*-transport stub may pass -(irrelevant on the default JSON transport). For a named-transport repo needing -a name outside the superset: regenerate with the JSON transport -(`wads-migrate ci-to-stub`), PR `wads.ci_secrets.DEFAULT_CI_SECRETS` (benefits -all repos), or use `--variable` if the value isn't sensitive. +(irrelevant on the opt-in JSON transport). For a named-transport repo needing +a name outside the superset: use `--variable` if the value isn't sensitive, +PR `wads.ci_secrets.DEFAULT_CI_SECRETS` (benefits all repos), or regenerate +with the JSON transport (`wads-migrate ci-to-stub --transport json`) on a repo +whose runs GitHub already accepts. Publishing runs only on the repo's **default branch**, when validation passes, when the commit isn't `[skip ci]`, and when `[tool.wads.ci.publish].enabled`. @@ -335,7 +339,7 @@ gh repo edit ORG/REPO --enable-discussions # enable when false - [ ] `[tool.wads.ci]` section present (or defaults are acceptable) - [ ] Secrets the code needs are configured via `wads-secrets add` (declares in `[tool.wads.ci.env]`; on a named-transport stub it also adds the - pass-through line — the default JSON transport needs none) + pass-through line — the opt-in JSON transport needs none) - [ ] `.github/workflows/ci.yml` is the stub (calls `uv-ci.yml@master`) OR the inline `github_ci_uv.yml` escape valve - [ ] `PYPI_PASSWORD` secret is a PyPI API token diff --git a/wads/licence_check.py b/wads/licence_check.py index 55ff26cc..49578f53 100644 --- a/wads/licence_check.py +++ b/wads/licence_check.py @@ -692,7 +692,7 @@ def declared_requirements( ... declared_requirements(pyproject={'project': { ... 'name': 'x', 'dynamic': ['dependencies']}}) ... except DetectorError as error: - ... print(str(error)[:59]) + ... print(str(error)[:58]) this project lists `dependencies` in [project].dynamic, so """ if pyproject is None: @@ -948,7 +948,11 @@ class LicencePolicy: allowed: tuple[str, ...] = DFLT_ALLOWED forbidden: tuple[str, ...] = DFLT_FORBIDDEN - exceptions: Mapping[str, str] = types.MappingProxyType({}) + # A factory, not a plain default: Python 3.11's dataclasses reject an + # unhashable default, and a mappingproxy is one there (i2mint/wads#100). + exceptions: Mapping[str, str] = dataclasses.field( + default_factory=lambda: types.MappingProxyType({}) + ) include_extras: tuple[str, ...] = () unknown_is_failure: bool = True unclassified_is_failure: bool = False @@ -986,7 +990,7 @@ def from_mapping(cls, config: Mapping[str, Any], /) -> "LicencePolicy": >>> try: ... LicencePolicy.from_mapping({'forbiden': ['GPL']}) ... except ValueError as error: - ... print(error) + ... print(error) # doctest: +NORMALIZE_WHITESPACE unknown [tool.wads.licence] key 'forbiden'; known keys are: allowed, enabled, exceptions, forbidden, include-extras, unclassified-is-failure, unknown-is-failure @@ -996,7 +1000,7 @@ def from_mapping(cls, config: Mapping[str, Any], /) -> "LicencePolicy": >>> try: ... LicencePolicy.from_mapping({'allow': ['MIT']}) ... except ValueError as error: - ... print(error) + ... print(error) # doctest: +NORMALIZE_WHITESPACE [tool.wads.licence] key 'allow' is from the earlier table shape: rename it to `allowed` (it pairs with `forbidden`, where `allow` had no counterpart) """ diff --git a/wads/migration.py b/wads/migration.py index 037d7e52..c4b3d589 100644 --- a/wads/migration.py +++ b/wads/migration.py @@ -584,7 +584,9 @@ def migrate_setuptools_to_hatching( pyproject_dict["project"]["name"] = required_fields["name"] pyproject_dict["project"]["version"] = required_fields["version"] pyproject_dict["project"]["description"] = required_fields["description"] - pyproject_dict["project"]["license"] = {"text": required_fields["license"]} + from wads.toml_util import pep639_license + + pyproject_dict["project"]["license"] = pep639_license(required_fields["license"]) if "urls" not in pyproject_dict["project"]: pyproject_dict["project"]["urls"] = {} @@ -865,7 +867,7 @@ def migrate_ci_to_stub( old_ci: Union[str, Path] = None, *, pin: str = "@master", - transport: str = "json", + transport: Optional[str] = None, trigger_mode: Optional[str] = None, run_ci_marker: Optional[str] = None, ) -> str: @@ -878,25 +880,29 @@ def migrate_ci_to_stub( `i2mint/wads/actions/read-ci-config` action. Args: - old_ci: Optional path or content of the existing CI workflow. Used only - to locate a nearby pyproject.toml when ``transport="named"``; - with the default JSON transport the stub is the same regardless of - what was there. + old_ci: Optional path or content of the existing CI workflow. Used to + locate a nearby pyproject.toml, whose ``[tool.wads.ci.env]`` decides + which secrets the default named transport passes. pin: The wads ref the stub points at. Defaults to ``"@master"`` (floats with wads). For release-sensitive repos, pin to a tag, e.g. ``pin="@0.2.15"`` (wads tags have no ``v`` prefix). With the - default JSON transport the pinned ref's ``uv-ci.yml`` must declare + JSON transport the pinned ref's ``uv-ci.yml`` must declare ``WADS_CI_SECRETS_JSON`` (releases after 0.2.14) — pinning an older tag produces a workflow GitHub rejects at parse time, so a - warning is emitted for any non-master pin. Must start with ``"@"``. - transport: ``"json"`` (default) passes the repo's whole secrets - context as one ``WADS_CI_SECRETS_JSON`` secret — any secret name - works, nothing to enumerate. ``"named"`` passes an explicit subset - (PYPI_PASSWORD + the [tool.wads.ci.env]-declared secrets) for - repos that want a minimal secret surface; every name must then be - in the frozen wads superset or GitHub rejects the workflow at - parse time (issue #63) — out-of-superset names trigger a loud - warning. + warning is emitted for any non-master JSON pin. Must start with + ``"@"``. + transport: ``None`` (default) keeps the transport of the stub at + ``old_ci`` when there is one, and otherwise uses ``"named"``, so a + re-render never silently changes what an existing repo runs. + ``"named"`` passes an explicit subset + (PYPI_PASSWORD + the [tool.wads.ci.env]-declared secrets); every + name must be in the frozen wads superset or GitHub rejects the + workflow at parse time (issue #63) — out-of-superset names trigger + a loud warning. ``"json"`` (opt-in) passes the repo's whole secrets + context as one ``WADS_CI_SECRETS_JSON`` secret, so any secret name + works, but GitHub's malicious-workflow scanner holds its runs on + new repositories (``action_required``, zero jobs; issues #74, #88), + which is why it is no longer the default. trigger_mode: ``"auto"`` or ``"on-demand"``. ``None`` (default) takes ``[tool.wads.ci.trigger].mode`` from the pyproject.toml of the repo holding ``old_ci`` (``"auto"`` when there is none). On-demand stubs @@ -912,19 +918,22 @@ def migrate_ci_to_stub( >>> stub = migrate_ci_to_stub() >>> 'i2mint/wads/.github/workflows/uv-ci.yml@master' in stub True - >>> 'WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}' in stub + >>> 'PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}' in stub True - >>> pinned = migrate_ci_to_stub(pin='@0.2.15') # warns on stderr + >>> 'toJSON(secrets)' in stub + False + >>> pinned = migrate_ci_to_stub(pin='@0.2.15') >>> 'uv-ci.yml@0.2.15' in pinned True - >>> named = migrate_ci_to_stub(transport='named') - >>> 'PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}' in named + >>> as_json = migrate_ci_to_stub(transport='json') + >>> 'WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}' in as_json True - >>> 'WADS_CI_SECRETS_JSON' in named - False """ if not pin.startswith("@"): raise ValueError(f"pin must start with '@', got {pin!r}") + auto_transport = transport is None + if transport is None: + transport = _existing_stub_transport(old_ci) if transport not in ("json", "named"): raise ValueError(f"transport must be 'json' or 'named', got {transport!r}") if transport == "json" and pin != "@master": @@ -942,36 +951,27 @@ def migrate_ci_to_stub( if pin != "@master": stub = stub.replace("uv-ci.yml@master", f"uv-ci.yml{pin}") if transport == "named": - from wads.ci_secrets import ( - render_stub_json_transport, - render_stub_secrets_passthrough, - ) + from wads.ci_secrets import stub_with_named_transport names = _stub_secret_names_for(old_ci) - _warn_named_transport_outside_superset(names) - # Swap the template's JSON-transport comment paragraph (the lines - # from "# Transport:" down to the JSON line) for a named-mode one, - # so the stub doesn't describe a transport it isn't using. - named_region = ( - " # Transport (NAMED, legacy): explicitly passes only the secrets\n" - " # listed below (PYPI_PASSWORD + those declared in\n" - " # [tool.wads.ci.env]). Every name must be in the frozen wads\n" - " # superset (wads/ci_secrets.py) or GitHub rejects the workflow\n" - " # at parse time. The default JSON transport has no such limit;\n" - " # regenerate with `wads-migrate ci-to-stub` to switch.\n" - " secrets:\n" + render_stub_secrets_passthrough(names) + "\n" - ) - json_line = render_stub_json_transport() - json_region = re.compile( - r"^ # Transport:.*?" + re.escape(json_line) + r"\n", - re.DOTALL | re.MULTILINE, - ) - stub, n_replaced = json_region.subn(named_region, stub) - if n_replaced != 1: - raise ValueError( - "stub template changed shape: could not locate the JSON " - "transport region to convert to named transport" + from wads.ci_secrets import DEFAULT_CI_SECRETS + + outside = [n for n in names if n not in DEFAULT_CI_SECRETS] + if auto_transport and outside: + # A named stub passing these could not start (issue #63). With no + # explicit choice, keep the JSON transport, which passes any name. + print( + f"note: using the JSON secrets transport because {outside} " + f"are outside the wads secrets superset, so a named stub " + f"passing them could not start (i2mint/wads#63). On a brand-new " + f"repo GitHub may hold JSON-transport runs (i2mint/wads#74); " + f"storing non-sensitive values as repository variables avoids " + f"both.", + file=sys.stderr, ) + else: + _warn_named_transport_outside_superset(names) + stub = stub_with_named_transport(stub, names) # Legacy templates carried a placeholder instead of a transport line. if "#SECRETS_BLOCK#" in stub: from wads.ci_secrets import render_stub_secrets_passthrough @@ -999,6 +999,17 @@ def migrate_ci_to_stub( return render_stub_inputs(stub, stub_inputs_of(old_ci)) +def _existing_stub_transport(old_ci) -> str: + """The secrets transport of the stub file at ``old_ci``; ``"named"`` otherwise.""" + from wads.ci_trigger import stub_shape + + if not old_ci: + return "named" + if "\n" not in str(old_ci) and os.path.isfile(str(old_ci)): + return stub_shape(Path(old_ci).read_text())["transport"] + return stub_shape(str(old_ci))["transport"] # content; "named" if not a stub + + def _stub_secret_names_for(old_ci) -> list: """Secret names a *named-transport* stub should pass, from nearby pyproject. @@ -1018,32 +1029,10 @@ def _stub_secret_names_for(old_ci) -> list: def _warn_named_transport_outside_superset(names) -> list: - """Warn loudly for names a named-transport stub cannot legally pass. + """Warn for names a named-transport stub cannot pass (see :mod:`wads.ci_secrets`).""" + from wads.ci_secrets import warn_names_outside_superset - A caller may only pass secrets the reusable workflow declares. With - ``transport="named"`` that universe is the frozen superset in - :data:`wads.ci_secrets.DEFAULT_CI_SECRETS`; a stub naming anything outside - it produces a workflow GitHub rejects at parse time — zero jobs, an opaque - ``startup_failure`` (issue #63). Returns the offending names. - """ - from wads.ci_secrets import DEFAULT_CI_SECRETS - - outside = [n for n in names if n not in DEFAULT_CI_SECRETS] - for name in outside: - print( - f"warning: {name!r} is not in the wads secrets superset, so a stub " - f"passing it by name CANNOT START (GitHub rejects the workflow at " - f"parse time with `startup_failure`). Either:\n" - f" - if the value is not actually sensitive, store it as a " - f"repository VARIABLE (`gh variable set {name}`) — declared env " - f"vars fall back to repo variables automatically; or\n" - f" - use the default JSON transport (`wads-migrate ci-to-stub` " - f"without --transport named), which passes every secret; or\n" - f" - keep the inline workflow (`wads-migrate ci-to-uv`, don't " - f"stub-ify).", - file=sys.stderr, - ) - return outside + return warn_names_outside_superset(names) def _find_pyproject_near(old_ci) -> Path | None: @@ -1244,7 +1233,7 @@ def main(): default="@master", help=( "wads ref to pin in the stub (default '@master'). " - "Use e.g. '@v0.1.81' to freeze." + "Use e.g. '@0.2.15' to freeze (tags have no 'v' prefix)." ), ) fleet_parser.add_argument( @@ -1286,9 +1275,9 @@ def main(): "else '@master' (floats with " "wads — convenient, occasional CI breakage on bad wads merges). " "Use e.g. '@0.2.15' to freeze (tags have no 'v' prefix). The " - "default JSON transport needs a ref whose uv-ci.yml declares " - "WADS_CI_SECRETS_JSON (releases after 0.2.14); for older pins " - "use --transport named. Must start with '@'." + "JSON transport needs a ref whose uv-ci.yml declares " + "WADS_CI_SECRETS_JSON (releases after 0.2.14); the named transport " + "works with any pin. Must start with '@'." ), ) stub_parser.add_argument( @@ -1297,12 +1286,13 @@ def main(): choices=("json", "named"), help=( "How the stub passes secrets to the reusable workflow (default: the " - "existing stub's, else json). 'json' " - "(default) serializes the repo's whole secrets context into one " - "WADS_CI_SECRETS_JSON secret — any secret name works. 'named' " - "passes an explicit subset (minimal secret surface), but every " - "name must be in the frozen wads superset or the workflow cannot " - "start." + "existing stub's, else named). 'named' passes PYPI_PASSWORD plus " + "the [tool.wads.ci.env]-declared secrets; every name must be in " + "the frozen wads superset or the workflow cannot start. 'json' " + "serializes the repo's whole secrets context into one " + "WADS_CI_SECRETS_JSON secret (any secret name works), but GitHub's " + "malicious-workflow scanner holds its runs on new repos " + "(action_required, zero jobs; i2mint/wads#74)." ), ) @@ -1494,7 +1484,9 @@ def main(): # (e.g. after changing [tool.wads.ci.trigger]) never silently unpins it. shape = stub_shape(existing) pin = args.pin or shape["pin"] - transport = args.transport or shape["transport"] + # None lets migrate_ci_to_stub keep an existing stub's transport, or + # choose named for a new one (JSON if a name is outside the superset). + transport = args.transport if classify_ci_workflow(existing) == "stub": try: _, dropped = stub_customizations(existing) @@ -1523,8 +1515,9 @@ def main(): if pin == "@master": print( "\nPinned to @master (floats with wads). If you need version " - "stability for this repo, re-run with `--pin @vX.Y.Z` " - "(latest wads tag visible via `gh release list -R i2mint/wads`).", + "stability for this repo, re-run with `--pin @X.Y.Z` (tags " + "have no 'v' prefix; latest via " + "`gh api repos/i2mint/wads/tags --jq '.[0].name'`).", file=sys.stderr, ) diff --git a/wads/pack.py b/wads/pack.py index d20184a7..bb598ebc 100644 --- a/wads/pack.py +++ b/wads/pack.py @@ -565,9 +565,12 @@ def extract_pkg_dir_and_name( `pkg_spec` can be an imported package (must be a locally developped package) whose name and containing directory is the same): - >>> import wads - >>> extract_pkg_dir_and_name(wads) # doctest: +ELLIPSIS - (.../wads', 'wads') + >>> import os, wads + >>> pkg_dir, pkg_name = extract_pkg_dir_and_name(wads) + >>> pkg_name + 'wads' + >>> os.path.isfile(os.path.join(pkg_dir, pkg_name, '__init__.py')) + True You can also just specify the name of the package (it will be imported): diff --git a/wads/populate.py b/wads/populate.py index 72551d2e..8b1261c5 100755 --- a/wads/populate.py +++ b/wads/populate.py @@ -227,8 +227,10 @@ def write_pyproject_configs(pkg_dir: str, configs: dict): if documentation_url: data["project"]["urls"]["Documentation"] = documentation_url - # Update license using inline table syntax - data["project"]["license"] = {"text": license_name} + # PEP 639 SPDX string when the name is valid SPDX, else the legacy table + from wads.toml_util import pep639_license + + data["project"]["license"] = pep639_license(license_name) # Add optional fields if present if configs.get("keywords"): @@ -1087,10 +1089,25 @@ def _add_ci_def( ci_def = render_minimal_env_placeholders(ci_def, name) + # A new repo's stub passes its secrets by NAME: the template's JSON + # transport gets held by GitHub's malicious-workflow scanner on new + # repositories (action_required, zero jobs; i2mint/wads#74, #88). + from wads.ci_secrets import ( + render_stub_json_transport, + stub_with_named_transport, + warn_names_outside_superset, + ) + + is_bundled_stub = os.path.abspath(ci_tpl_path) == os.path.abspath( + github_ci_uv_stub_path + ) + if is_bundled_stub and render_stub_json_transport() in ci_def: + names = ci_config.stub_secret_names() if ci_config else ["PYPI_PASSWORD"] + warn_names_outside_superset(names) + ci_def = stub_with_named_transport(ci_def, names) + # Legacy templates carried a #SECRETS_BLOCK# placeholder (per-repo - # named transport). The current stub template passes the whole secrets - # context as one WADS_CI_SECRETS_JSON secret and has no placeholder, - # so this branch only fires for old templates. + # named transport); this branch only fires for those. if "#SECRETS_BLOCK#" in ci_def: from wads.ci_secrets import render_stub_secrets_passthrough @@ -1256,7 +1273,8 @@ def _get_org_slash_proj(repo: str) -> str: >>> _get_org_slash_proj('https://github.com/thorwhalen/ut/') 'thorwhalen/ut' """ - *_, org, proj_name = ensure_no_slash_suffix(repo).split("/") + # A URL separator, not os.sep (ensure_no_slash_suffix strips "\\" on Windows) + *_, org, proj_name = repo.rstrip("/").split("/") return f"{org}/{proj_name}" diff --git a/wads/scripts/read_ci_config.py b/wads/scripts/read_ci_config.py index 8187d3ce..e6018392 100755 --- a/wads/scripts/read_ci_config.py +++ b/wads/scripts/read_ci_config.py @@ -129,6 +129,22 @@ def read_and_export_ci_config(pyproject_path: str | Path = ".") -> int: _set_output("env-defaults", config.env_vars_defaults) _set_output("env-aliases", config.env_secret_aliases) + # Declared test tooling that CI will not install (i2mint/wads#59). + # A warning only: installing it automatically would change what every + # repo without an explicit `extras` setting runs. + try: + uninstalled = config.uninstalled_test_extras + except Exception: # a diagnostic must never fail the setup job + uninstalled = {} + for extra, packages in uninstalled.items(): + print( + f"::warning title=Test extra not installed::The '{extra}' extra " + f"({', '.join(packages)}) is not installed in CI, because " + f"[tool.wads.ci.install].extras is unset (CI installs core " + f'dependencies only). Set extras = "{extra}" to install it, ' + f'or extras = "" to keep core-only and silence this warning.' + ) + # Print summary print("[OK] CI configuration loaded successfully") print(f" Project: {config.project_name}") diff --git a/wads/secrets_cli.py b/wads/secrets_cli.py index 8c7cd0d3..48f8d8a1 100644 --- a/wads/secrets_cli.py +++ b/wads/secrets_cli.py @@ -6,10 +6,11 @@ 1. **pyproject** ``[tool.wads.ci.env]`` — declares the env var (and whether it is required), so the reusable workflow exports it into the job environment. 2. **transport** — the repo's ``ci.yml`` stub passes secrets to the reusable - workflow. Modern stubs pass the whole secrets context as one - ``WADS_CI_SECRETS_JSON`` secret, so *no per-secret stub edit is needed*; - legacy named-transport stubs list each secret explicitly (and every listed - name must be in the frozen wads superset). + workflow. Named-transport stubs (the default for new stubs) list each + secret explicitly, and every listed name must be in the frozen wads + superset; ``add`` inserts the line. Stubs on the opt-in JSON transport pass + the whole secrets context as one ``WADS_CI_SECRETS_JSON`` secret, so *no + per-secret stub edit is needed* there. ``wads-secrets add`` performs the needed edits in one step, and can also set the secret's value on GitHub via ``gh`` — so a single command takes a secret @@ -303,8 +304,9 @@ def add( f"name — passing this one would make the workflow FAIL TO START, " f"so ci.yml was NOT edited (declared in pyproject only). Either " f"regenerate the stub with the JSON transport " - f"(`wads-migrate ci-to-stub`), which passes every secret; or, if " - f"the value is not sensitive, use " + f"(`wads-migrate ci-to-stub --transport json`), which passes every " + f"secret (GitHub may hold its runs on a brand-new repo, " + f"i2mint/wads#74); or, if the value is not sensitive, use " f"`wads-secrets add {var_name} --variable` instead." ) else: diff --git a/wads/tests/data/golden/python_lib/.github/workflows/ci.yml b/wads/tests/data/golden/python_lib/.github/workflows/ci.yml index a09825a3..84179a75 100644 --- a/wads/tests/data/golden/python_lib/.github/workflows/ci.yml +++ b/wads/tests/data/golden/python_lib/.github/workflows/ci.yml @@ -7,9 +7,9 @@ # Pinning: `@master` floats with wads. If you need version stability for # a release-sensitive repo, change `@master` to a wads tag (e.g. `@0.2.15`; # tags have no `v` prefix). A stub whose `secrets:` block passes the JSON -# transport (the default below) needs a tag from a release after 0.2.14 — -# older tags don't declare that secret and GitHub then rejects the -# workflow at parse time. +# transport (`WADS_CI_SECRETS_JSON`) needs a tag from a release after +# 0.2.14 — older tags don't declare that secret and GitHub then rejects +# the workflow at parse time. The named transport works with any tag. # CI failure does not block a published release — it blocks the publish # step itself — so floating master is generally safe. # @@ -33,24 +33,20 @@ jobs: permissions: contents: write pages: write - # Transport: this repo's whole `secrets` context, serialized into the one - # secret the reusable workflow declares. Double-encoded (toJSON twice) so - # the value is a single line — a multiline secret would register its `{` - # and `}` lines as global log masks. Any secret name works; there is no - # fixed list to fall outside of. (Cross-owner `secrets: inherit` does not - # propagate secrets, so it cannot replace this.) + # Transport (NAMED, the default): passes only the secrets listed below -- + # PYPI_PASSWORD plus the backing secret of each env var declared in + # [tool.wads.ci.env] (`wads-secrets add VAR_NAME` updates both). Every + # name must be in the frozen wads superset (wads/ci_secrets.py) or GitHub + # rejects the workflow at parse time. # - # Note this hands EVERY secret this repo can read — including org-level - # ones — to the called workflow. For a minimal secret surface (only the - # names you list), regenerate with - # wads-migrate ci-to-stub --transport named + # The opt-in JSON transport (`wads-migrate ci-to-stub --transport json`) + # passes every secret without a list, but GitHub's malicious-workflow + # scanner holds runs that use it on new repositories: `action_required`, + # zero jobs, no log (i2mint/wads#74). # - # *Which* of these become job env vars — and which are required — is - # driven entirely by [tool.wads.ci.env] in pyproject.toml; nothing is - # exported unless declared there (`wads-secrets add VAR_NAME` declares - # one and can set its value). Non-sensitive values don't need a secret: - # use [tool.wads.ci.env].defaults (committed literals) or a repository - # *variable* (`gh variable set NAME`) — declared names fall back to - # repo variables automatically. + # *Which* of these become job env vars -- and which are required -- is + # driven by [tool.wads.ci.env] in pyproject.toml. Non-sensitive values + # don't need a secret: use [tool.wads.ci.env].defaults or a repository + # *variable* (`gh variable set NAME`). secrets: - WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }} + PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }} diff --git a/wads/tests/data/golden/python_lib/pyproject.toml b/wads/tests/data/golden/python_lib/pyproject.toml index c61e6e07..52122eee 100644 --- a/wads/tests/data/golden/python_lib/pyproject.toml +++ b/wads/tests/data/golden/python_lib/pyproject.toml @@ -10,15 +10,13 @@ version = "1.2.3" description = "Test package" readme = "README.md" requires-python = ">=3.10" +license = "MIT" keywords = [] authors = [ { name = "John Doe" }, ] dependencies = [] -[project.license] -text = "mit" - [project.urls] Homepage = "https://github.com/myorg/mypkg" Repository = "https://github.com/myorg/mypkg" diff --git a/wads/tests/test_ci_install_extras_warning.py b/wads/tests/test_ci_install_extras_warning.py new file mode 100644 index 00000000..83ef26b0 --- /dev/null +++ b/wads/tests/test_ci_install_extras_warning.py @@ -0,0 +1,105 @@ +"""i2mint/wads#59: warn when CI never installs a repo's declared test extra. + +``[tool.wads.ci.install].extras`` defaults to empty, so CI installs core +dependencies only. A repo that keeps test tooling in a ``dev``/``test`` extra +and never sets ``extras`` gets a job that silently lacks it (a missing +``pytest-asyncio`` even turns async tests into skips). Installing such extras +automatically would change every repo's CI at once, which is the owner's call, +so this adds the additive half: a loud ``::warning::`` in every CI log. +""" + +import textwrap + +import pytest + +from wads.ci_config import CIConfig +from wads.scripts.read_ci_config import read_and_export_ci_config + + +def _config(optional_deps: dict, install: dict | None = None) -> CIConfig: + data = {"project": {"name": "mypkg", "optional-dependencies": optional_deps}} + if install is not None: + data["tool"] = {"wads": {"ci": {"install": install}}} + return CIConfig(data) + + +def test_a_dev_extra_with_real_tooling_is_reported_when_extras_is_unset(): + config = _config({"dev": ["pytest", "httpx>=0.27", "pytest-asyncio"]}) + assert config.uninstalled_test_extras == {"dev": ["httpx", "pytest-asyncio"]} + + +def test_tools_ci_provides_on_its_own_are_not_reported(): + """run-tests-uv installs pytest (+ pytest-cov); ruff comes from its action.""" + config = _config({"test": ["pytest>=7", "pytest-cov", "Ruff", "coverage[toml]"]}) + assert config.uninstalled_test_extras == {} + + +@pytest.mark.parametrize("install", [{"extras": ""}, {"extras": "dev"}, {"extras": []}]) +def test_an_explicit_extras_setting_is_respected(install): + """``extras = ""`` is the documented opt-out; any explicit value is a choice.""" + config = _config({"dev": ["httpx"]}, install=install) + assert config.uninstalled_test_extras == {} + + +def test_only_conventional_test_extra_names_count(): + config = _config({"docs": ["sphinx"], "gpu": ["torch"], "Testing": ["hypothesis"]}) + assert config.uninstalled_test_extras == {"Testing": ["hypothesis"]} + + +def test_self_references_and_unparsable_entries_do_not_crash(): + config = _config({"dev": ["mypkg[test]", "not a requirement !!", "numpy"]}) + assert config.uninstalled_test_extras == {"dev": ["numpy"]} + + +def test_read_ci_config_prints_a_github_warning(tmp_path, monkeypatch, capsys): + for var in ("GITHUB_OUTPUT", "GITHUB_ENV", "GITHUB_STEP_SUMMARY"): + monkeypatch.delenv(var, raising=False) + (tmp_path / "pyproject.toml").write_text( + textwrap.dedent( + """\ + [project] + name = "mypkg" + version = "0.1.0" + + [project.optional-dependencies] + dev = ["pytest", "httpx"] + """ + ) + ) + assert read_and_export_ci_config(tmp_path) == 0 + out = capsys.readouterr().out + warning = [line for line in out.splitlines() if line.startswith("::warning")] + assert len(warning) == 1 + assert "httpx" in warning[0] and "extras" in warning[0] and "dev" in warning[0] + + +def test_read_ci_config_is_quiet_when_nothing_is_missing(tmp_path, monkeypatch, capsys): + for var in ("GITHUB_OUTPUT", "GITHUB_ENV", "GITHUB_STEP_SUMMARY"): + monkeypatch.delenv(var, raising=False) + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "mypkg"\nversion = "0.1.0"\n' + '[project.optional-dependencies]\ndev = ["pytest"]\n' + ) + assert read_and_export_ci_config(tmp_path) == 0 + assert "::warning" not in capsys.readouterr().out + + +@pytest.mark.parametrize( + "optional_deps", + [ + ["dev"], # not a table + {"dev": "httpx"}, # a string, not a list + {"dev": [1, None, {"x": 1}, "httpx"]}, # non-string entries + {"dev": None}, + ], +) +def test_malformed_optional_dependencies_never_crash(optional_deps): + """A diagnostic must never be what fails the CI setup job.""" + config = _config(optional_deps) + result = config.uninstalled_test_extras + assert result in ({}, {"dev": ["httpx"]}) + + +def test_marker_gated_requirements_that_do_not_apply_are_not_reported(): + config = _config({"dev": ['httpx; python_version < "3.0"', "numpy"]}) + assert config.uninstalled_test_extras == {"dev": ["numpy"]} diff --git a/wads/tests/test_ci_trigger.py b/wads/tests/test_ci_trigger.py index 80883c98..5297e493 100644 --- a/wads/tests/test_ci_trigger.py +++ b/wads/tests/test_ci_trigger.py @@ -353,7 +353,7 @@ def test_the_stub_template_carries_each_anchor_once(): def test_auto_renders_the_template_byte_for_byte(): assert render_stub_trigger(STUB_TEMPLATE) == STUB_TEMPLATE - assert migrate_ci_to_stub() == STUB_TEMPLATE + assert migrate_ci_to_stub(transport="json") == STUB_TEMPLATE def test_on_demand_stub_says_nothing_runs_unless_asked(): diff --git a/wads/tests/test_collection_scope.py b/wads/tests/test_collection_scope.py new file mode 100644 index 00000000..b585e9a1 --- /dev/null +++ b/wads/tests/test_collection_scope.py @@ -0,0 +1,61 @@ +"""wads's own CI collects its package doctests, not only ``wads/tests``. + +The ``run-tests-uv`` action runs ``pytest --doctest-modules`` with no path, so +``[tool.pytest.ini_options].testpaths`` alone decides what is collected +(i2mint/wads#56). With ``testpaths = ["wads/tests"]`` every doctest in the +package was skipped in CI while it stayed green, and two of them had drifted +into failing. These tests pin the scope so it cannot silently shrink again. +""" + +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + +pytestmark = pytest.mark.skipif( + not (REPO_ROOT / "pyproject.toml").is_file(), + reason="needs a source checkout (runs pytest's collection on the repo)", +) + + +@pytest.fixture(scope="module") +def collected_ids(): + """Node ids CI's pathless ``pytest --doctest-modules`` would collect.""" + result = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "--collect-only", + "-q", + "--doctest-modules", + "-p", + "no:cacheprovider", + ], + cwd=REPO_ROOT, + capture_output=True, + text=True, + timeout=300, + ) + assert result.returncode == 0, result.stdout[-3000:] + result.stderr[-3000:] + return [line for line in result.stdout.splitlines() if "::" in line] + + +def test_package_doctests_are_collected(collected_ids): + assert "wads/licence_check.py::wads.licence_check.LicencePolicy.from_mapping" in ( + collected_ids + ) + + +def test_test_modules_are_still_collected(collected_ids): + assert any( + i.startswith("wads/tests/test_licence_check.py::") for i in collected_ids + ) + + +def test_templates_under_data_are_not_collected(collected_ids): + """``wads/data`` holds templates (e.g. ``test_smoke_tpl.py``), not Python.""" + assert not [i for i in collected_ids if i.startswith("wads/data/")] diff --git a/wads/tests/test_git_commit_push_retry.py b/wads/tests/test_git_commit_push_retry.py index b8d9fd61..be23fa60 100644 --- a/wads/tests/test_git_commit_push_retry.py +++ b/wads/tests/test_git_commit_push_retry.py @@ -358,6 +358,165 @@ def test_real_content_merged_concurrently_survives_the_version_bump_replay( assert 'version = "0.0.3"' in shown assert not (clone / ".git" / "rebase-merge").exists() + @pytest.mark.parametrize( + "filename, template", + [ + ("pyproject.toml", 'version="{}"\n'), # valid TOML, no spaces + ("pyproject.toml", "version = '{}'\n"), # single-quoted literal + ("pyproject.toml", ' version = "{}"\n'), # indented, padded + ("setup.cfg", "[metadata]\nversion={}\n"), + ], + ) + def test_version_bump_replay_tolerates_version_line_formatting( + self, remote_and_clone, tmp_path, filename, template + ): + """i2mint/wads#98: a differently-formatted version line must still be bumped. + + The replay used to rewrite the version with a ``sed`` that matched only + ``version = "X.Y.Z"`` (one space each side, double quotes). Any other + valid spelling made it a silent no-op, so git kept the concurrent + run's OLDER version while PyPI had this run's newer one (the #83 + desync, with no error logged). The rewrite must keep the file's own + formatting and change only the version number. + """ + origin, clone = remote_and_clone + _commit(clone, filename, template.format("0.0.4"), "**CI** bump to 0.0.4") + other = tmp_path / "earlier-release" + _git("clone", str(origin), str(other), cwd=tmp_path) + _configure(other) + _commit(other, filename, template.format("0.0.3"), "**CI** bump to 0.0.3") + _git("push", "origin", DEFAULT_BRANCH, cwd=other) + + result = run_push_step(clone) + + assert result.returncode == 0, result.stdout + result.stderr + shown = _git("show", f"origin/{DEFAULT_BRANCH}:{filename}", cwd=clone).stdout + assert shown == template.format("0.0.4") + assert _subjects(clone, f"origin/{DEFAULT_BRANCH}")[:2] == [ + "**CI** bump to 0.0.4", + "**CI** bump to 0.0.3", + ] + assert not (clone / ".git" / "rebase-merge").exists() + + def test_only_the_first_version_line_is_rewritten(self, remote_and_clone, tmp_path): + """Another table's ``version`` key (a tool's own setting) is left alone.""" + origin, clone = remote_and_clone + tail = '\n[tool.other]\nversion = "9.9.9"\n' + _commit(clone, "pyproject.toml", 'version = "0.0.4"\n' + tail, "**CI** bump") + other = tmp_path / "earlier-release" + _git("clone", str(origin), str(other), cwd=tmp_path) + _configure(other) + _commit( + other, + "pyproject.toml", + 'version = "0.0.3"\n' + tail + "\n[tool.added]\nx = 1\n", + "**CI** bump to 0.0.3, plus a merged table", + ) + _git("push", "origin", DEFAULT_BRANCH, cwd=other) + + result = run_push_step(clone) + + assert result.returncode == 0, result.stdout + result.stderr + shown = _git( + "show", f"origin/{DEFAULT_BRANCH}:pyproject.toml", cwd=clone + ).stdout + assert shown == 'version = "0.0.4"\n' + tail + "\n[tool.added]\nx = 1\n" + + @pytest.mark.parametrize( + "filename, template", + [ + ( + "pyproject.toml", + '[tool.other]\nversion = "9.9.9"\n\n[project]\nname = "p"\n' + 'version = "{}"\n\n[project.urls]\nHomepage = "x"\n', + ), + ( + "setup.cfg", + "[bumpversion]\nversion = 9.9.9\n\n[metadata]\nversion = {}\n", + ), + ( + "pyproject.toml", + '[[tool.a]]\nversion = "9.9.9"\n\n[ project ] # the package\n' + 'name = "p"\nmatrix = [\n ["x"],\n]\nversion = "{}"\n\n' + '[[tool.b]]\nversion = "8.8.8"\n', + ), + ( + "setup.cfg", + "[options]\nversion = 9.9.9\n\n[metadata] ; pkg\nversion = {}\n", + ), + ], + ) + def test_the_project_version_is_bumped_even_after_another_tables_version( + self, remote_and_clone, tmp_path, filename, template + ): + """Only [project] (pyproject) / [metadata] (setup.cfg) holds the version.""" + origin, clone = remote_and_clone + _commit(clone, filename, template.format("0.0.4"), "**CI** bump to 0.0.4") + other = tmp_path / "earlier-release" + _git("clone", str(origin), str(other), cwd=tmp_path) + _configure(other) + _commit(other, filename, template.format("0.0.3"), "**CI** bump to 0.0.3") + _git("push", "origin", DEFAULT_BRANCH, cwd=other) + + result = run_push_step(clone) + + assert result.returncode == 0, result.stdout + result.stderr + shown = _git("show", f"origin/{DEFAULT_BRANCH}:{filename}", cwd=clone).stdout + assert shown == template.format("0.0.4") + + def test_a_version_that_cannot_be_written_back_fails_loudly( + self, remote_and_clone, tmp_path + ): + """i2mint/wads#98: never push a replay that silently kept the old version. + + Upstream switched to a dynamic version (no ``version =`` line left), so + there is nowhere to write the version this run published. Pushing + anyway would record a version in git that disagrees with PyPI; the + step must stop with an error that names the file and the version. + """ + origin, clone = remote_and_clone + _commit(clone, "pyproject.toml", 'version = "0.0.3"\n', "**CI** bump to 0.0.3") + other = tmp_path / "went-dynamic" + _git("clone", str(origin), str(other), cwd=tmp_path) + _configure(other) + _commit(other, "pyproject.toml", 'dynamic = ["version"]\n', "go dynamic") + _git("push", "origin", DEFAULT_BRANCH, cwd=other) + + result = run_push_step(clone) + + assert result.returncode == 1 + assert "::error::" in result.stdout + assert "0.0.3" in result.stdout and "pyproject.toml" in result.stdout + assert _subjects(clone, f"origin/{DEFAULT_BRANCH}")[0] == "go dynamic" + assert not (clone / ".git" / "rebase-merge").exists() + assert not (clone / ".git" / "rebase-apply").exists() + + def test_a_version_file_deleted_upstream_fails_with_a_readable_error( + self, remote_and_clone, tmp_path + ): + """i2mint/wads#98 (UX note): modify/delete conflicts get the step's own error. + + ``git checkout --ours`` has no side to take when upstream deleted the + file, and used to abort the step under ``set -e`` with git's bare + "does not have our version", leaving a rebase in progress. + """ + origin, clone = remote_and_clone + _commit(clone, "pyproject.toml", 'version = "0.0.3"\n', "**CI** bump to 0.0.3") + other = tmp_path / "deleted" + _git("clone", str(origin), str(other), cwd=tmp_path) + _configure(other) + _git("rm", "-q", "pyproject.toml", cwd=other) + _git("commit", "-m", "drop pyproject", cwd=other) + _git("push", "origin", DEFAULT_BRANCH, cwd=other) + + result = run_push_step(clone) + + assert result.returncode == 1 + assert "::error::" in result.stdout + assert "pyproject.toml" in result.stdout + assert not (clone / ".git" / "rebase-merge").exists() + assert not (clone / ".git" / "rebase-apply").exists() + def test_conflicting_replay_fails_cleanly(self, remote_and_clone, tmp_path): """A conflict outside the version files aborts rather than wedging the repo. diff --git a/wads/tests/test_licence_check.py b/wads/tests/test_licence_check.py index 18a287b4..e238cbf2 100644 --- a/wads/tests/test_licence_check.py +++ b/wads/tests/test_licence_check.py @@ -1553,3 +1553,36 @@ def test_the_readme_worked_example_is_a_superset_of_the_defaults(): # And the resulting policy is one the tool would actually agree to run. checked = lc.self_check(policy) assert checked.uncovered_families == () + + +def test_dataclass_defaults_are_hashable_so_the_module_imports_on_py311(): + """i2mint/wads#100: Python 3.11 rejects unhashable dataclass defaults. + + ``dataclasses`` on 3.11 raises at class creation for any default whose + type is unhashable, so ``import wads.licence_check`` died there (a + ``mappingproxy`` default: hashable as a type only from 3.12, never + hashable over a ``dict``). CI tests 3.10 and 3.12, which both import fine, + so the rule is checked directly here on whatever interpreter runs it. + """ + import dataclasses + + offenders = [] + for name, obj in vars(lc).items(): + if not (isinstance(obj, type) and dataclasses.is_dataclass(obj)): + continue + for field in dataclasses.fields(obj): + if field.default is dataclasses.MISSING: + continue + try: + hash(field.default) + except TypeError: + offenders.append(f"{name}.{field.name}") + assert not offenders, f"unhashable dataclass defaults (break on 3.11): {offenders}" + + +def test_licence_policy_default_exceptions_are_empty_and_read_only(): + """The default-factory fix keeps the default an empty, read-only mapping.""" + policy = lc.LicencePolicy() + assert dict(policy.exceptions) == {} + with pytest.raises(TypeError): + policy.exceptions["x"] = "y" diff --git a/wads/tests/test_light_install.py b/wads/tests/test_light_install.py index 1fd12872..84a29c1c 100644 --- a/wads/tests/test_light_install.py +++ b/wads/tests/test_light_install.py @@ -67,9 +67,14 @@ def light_environment(): yield finally: sys.meta_path.remove(blocker) - # Re-import is left to other tests; restore originals to avoid surprises. - for m, mod in purged.items(): - sys.modules.setdefault(m, mod) + # Put sys.modules back EXACTLY: drop the fresh wads modules the test + # imported, then restore the originals. `setdefault` alone kept the + # fresh `wads` package object, which lacks attributes for submodules + # imported before the purge; on Python 3.10, mock.patch resolves + # "wads.project_setup.x" by attribute lookup from `wads` and failed. + for m in [m for m in sys.modules if m.startswith("wads")]: + del sys.modules[m] + sys.modules.update(purged) @pytest.mark.parametrize("module", LIGHT_MODULES) diff --git a/wads/tests/test_migration_rough_edges.py b/wads/tests/test_migration_rough_edges.py new file mode 100644 index 00000000..378be328 --- /dev/null +++ b/wads/tests/test_migration_rough_edges.py @@ -0,0 +1,98 @@ +"""i2mint/wads#52: rough edges in ``setup-to-pyproject`` output and pin hints.""" + +import re +from pathlib import Path + +import pytest + +from wads.ci_config import CIConfig +from wads.migration import migrate_setuptools_to_hatching +from wads.toml_util import pep639_license + +try: + import tomllib +except ModuleNotFoundError: # Python 3.10 + import tomli as tomllib + +MIGRATION_SOURCE = (Path(__file__).parents[1] / "migration.py").read_text() + + +def _migrated_license(license_value): + cfg = { + "metadata": { + "name": "myproj", + "version": "0.1.0", + "description": "My project", + "url": "https://github.com/user/myproj", + "license": license_value, + } + } + return tomllib.loads(migrate_setuptools_to_hatching(cfg))["project"]["license"] + + +@pytest.mark.parametrize( + "given, expected", + [ + ("MIT", "MIT"), + ("mit", "MIT"), + ("Apache Software License", "Apache-2.0"), + ("apache-2.0", "Apache-2.0"), + ], +) +def test_setup_to_pyproject_emits_an_spdx_license_string(given, expected): + """Item 1: PEP 639 ``license = ""``, not the deprecated table.""" + assert _migrated_license(given) == expected + + +def test_a_license_that_is_not_spdx_keeps_the_table_form(): + """Hatchling rejects a non-SPDX ``license`` string, so never emit one.""" + assert _migrated_license("Proprietary, all rights reserved") == { + "text": "Proprietary, all rights reserved" + } + + +def test_pep639_license_falls_back_without_a_new_enough_packaging(monkeypatch): + import builtins + + real_import = builtins.__import__ + + def no_licenses(name, *args, **kwargs): + if name == "packaging.licenses": + raise ImportError(name) + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", no_licenses) + assert pep639_license("MIT") == {"text": "MIT"} + + +def test_an_empty_ci_project_name_falls_back_to_the_package_name(): + """Item 2: ``project_name = ""`` never reaches ``--cov=`` as an empty target.""" + config = CIConfig( + {"project": {"name": "myproj"}, "tool": {"wads": {"ci": {"project_name": ""}}}} + ) + assert config.project_name == "myproj" + + +def test_pin_hints_use_bare_version_tags(): + """Items 4/5: wads tags have no ``v`` prefix, and wads publishes tags, not releases.""" + assert not re.findall(r"@v\d", MIGRATION_SOURCE) + assert "@vX" not in MIGRATION_SOURCE + assert "gh release list" not in MIGRATION_SOURCE + + +def test_org_slash_proj_strips_a_url_slash_even_on_windows(monkeypatch): + """A URL's trailing ``/`` is not an OS path separator (``\\`` on Windows).""" + import wads.util + from wads.populate import _get_org_slash_proj + + monkeypatch.setattr(wads.util, "path_sep", "\\") + assert _get_org_slash_proj("https://github.com/thorwhalen/ut/") == "thorwhalen/ut" + + +@pytest.mark.parametrize("value", [None, 3]) +def test_pep639_license_passes_non_strings_through_as_a_table(value): + assert pep639_license(value) == {"text": value} + + +def test_pep639_license_leaves_a_table_alone(): + assert pep639_license({"text": "MIT"}) == {"text": "MIT"} diff --git a/wads/tests/test_readme.py b/wads/tests/test_readme.py new file mode 100644 index 00000000..b5d21c46 --- /dev/null +++ b/wads/tests/test_readme.py @@ -0,0 +1,46 @@ +"""The README's agent-facing example runs, and its local links resolve.""" + +import re +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +README = REPO_ROOT / "README.md" + +pytestmark = pytest.mark.skipif( + not README.is_file(), reason="needs a source checkout with README.md" +) + + +def _section(text: str, heading: str) -> str: + start = text.index(heading) + nxt = text.find("\n## ", start + len(heading)) + return text[start : nxt if nxt != -1 else None] + + +def test_the_agent_example_runs(): + """The python block under "For AI agents" is executed as written.""" + section = _section(README.read_text(), "## For AI agents") + (block,) = re.findall(r"```python\n(.*?)```", section, re.DOTALL) + exec(compile(block, "README.md:For AI agents", "exec"), {}) + + +def test_relative_links_point_at_existing_files(): + text = README.read_text() + targets = re.findall(r"\]\(([^)#\s]+)\)", text) + local = [t for t in targets if not re.match(r"[a-z]+:", t)] + missing = [t for t in local if not (REPO_ROOT / t).exists()] + assert local, "expected some relative links to check" + assert not missing, f"README links to missing files: {missing}" + + +def test_in_page_anchors_match_a_heading(): + text = README.read_text() + slugs = { + re.sub(r"[^\w\- ]", "", h.strip().lower()).replace(" ", "-") + for h in re.findall(r"^#+ (.+)$", text, re.MULTILINE) + } + anchors = re.findall(r"\]\(#([^)]+)\)", text) + assert "for-carbon-based-contributors" in anchors + assert not [a for a in anchors if a not in slugs] diff --git a/wads/tests/test_secrets_cli.py b/wads/tests/test_secrets_cli.py index b79dce0c..b0dbbb46 100644 --- a/wads/tests/test_secrets_cli.py +++ b/wads/tests/test_secrets_cli.py @@ -34,13 +34,13 @@ def _make_repo(tmp_path, stub_text): @pytest.fixture def repo(tmp_path): - """A repo with the default (JSON-transport) stub.""" - return _make_repo(tmp_path, migrate_ci_to_stub()) + """A repo with a JSON-transport stub (opt-in since i2mint/wads#74).""" + return _make_repo(tmp_path, migrate_ci_to_stub(transport="json")) @pytest.fixture def named_repo(tmp_path): - """A repo with a legacy named-transport stub.""" + """A repo with a named-transport stub (the default).""" return _make_repo(tmp_path, migrate_ci_to_stub(transport="named")) @@ -180,6 +180,9 @@ def test_add_on_named_stub_refuses_outside_superset_edit(named_repo, capsys): out = capsys.readouterr().out assert "superset" in out and "FAIL TO START" in out and "NOT edited" in out assert "--variable" in out # points at the non-sensitive-value remedy + # ...and at a command that really switches transport: a bare `ci-to-stub` + # keeps the existing stub's (named) transport. + assert "--transport json" in out def test_add_variable_on_inline_workflow_warns(tmp_path, capsys): @@ -222,7 +225,7 @@ def test_add_variable_skips_transport_and_superset(repo, capsys): def test_json_stub_pinned_to_old_tag_warns(capsys): """Reviewer finding F2: a JSON stub pinned to a pre-JSON tag cannot start; migrate_ci_to_stub must warn on any non-master pin with json transport.""" - stub = migrate_ci_to_stub(pin="@0.2.14") + stub = migrate_ci_to_stub(pin="@0.2.14", transport="json") assert "uv-ci.yml@0.2.14" in stub err = capsys.readouterr().err assert "CANNOT START" in err and "--transport named" in err diff --git a/wads/tests/test_stub_transport_default.py b/wads/tests/test_stub_transport_default.py new file mode 100644 index 00000000..405fd475 --- /dev/null +++ b/wads/tests/test_stub_transport_default.py @@ -0,0 +1,240 @@ +"""New stubs pass secrets by NAME by default; the JSON transport is opt-in. + +i2mint/wads#74 and #88: a stub whose ``secrets:`` block is +``WADS_CI_SECRETS_JSON: ${{ toJSON(toJSON(secrets)) }}`` serialises the whole +secrets context into a workflow in another repository, which is what a +secret-exfiltration workflow looks like. GitHub's malicious-workflow scanner +holds such runs on NEW repositories: ``action_required``, zero jobs, no log, +not approvable through the API. It was reproduced on four new repos +(tituli, looks, mergeset, acquaint), and switching to the named transport made +the next push start every time. So everything that writes a NEW stub +(``populate``, ``wads-migrate ci-to-stub`` on an inline workflow) now defaults +to named, while an existing stub keeps the transport it already has. +""" + +import subprocess + +import pytest +import yaml + +from wads import github_ci_uv_stub_path +from wads.ci_secrets import render_stub_json_transport, stub_with_named_transport +from wads.ci_trigger import stub_shape +from wads.migration import migrate_ci_to_stub +from wads.populate import populate_pkg_dir + +JSON_LINE = render_stub_json_transport().strip() +PYPI_LINE = "PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}" +STUB_TEMPLATE = open(github_ci_uv_stub_path).read() + + +def _secrets_of(stub_text: str) -> dict: + return yaml.safe_load(stub_text)["jobs"]["ci"]["secrets"] + + +def test_migrate_ci_to_stub_defaults_to_the_named_transport(): + stub = migrate_ci_to_stub() + assert "toJSON(secrets)" not in stub + assert _secrets_of(stub) == {"PYPI_PASSWORD": "${{ secrets.PYPI_PASSWORD }}"} + + +def test_the_json_transport_is_still_available_as_an_opt_in(): + stub = migrate_ci_to_stub(transport="json") + assert stub == STUB_TEMPLATE + assert JSON_LINE in stub + + +def test_a_non_stub_workflow_converts_to_the_named_transport(): + assert stub_shape(None)["transport"] == "named" + assert stub_shape("name: CI\non: [push]\njobs: {}\n")["transport"] == "named" + + +def test_an_existing_json_stub_keeps_its_transport_on_rerender(): + """Re-rendering never silently changes the transport a repo already runs.""" + assert stub_shape(STUB_TEMPLATE)["transport"] == "json" + + +def test_the_named_stub_says_why_json_is_not_the_default(): + """A reader who opts into JSON must be told about the held-run failure.""" + stub = migrate_ci_to_stub() + assert "action_required" in stub + assert "--transport json" in stub + assert "legacy" not in stub.lower() + + +def test_stub_with_named_transport_lists_the_given_names(): + stub = stub_with_named_transport(STUB_TEMPLATE, ["PYPI_PASSWORD", "HF_TOKEN"]) + assert _secrets_of(stub) == { + "PYPI_PASSWORD": "${{ secrets.PYPI_PASSWORD }}", + "HF_TOKEN": "${{ secrets.HF_TOKEN }}", + } + + +def test_stub_with_named_transport_rejects_a_stub_without_the_json_region(): + with pytest.raises(ValueError, match="JSON transport region"): + stub_with_named_transport("name: CI\n", ["PYPI_PASSWORD"]) + + +@pytest.fixture +def populated_ci(tmp_path): + pkg_dir = tmp_path / "mypkg" + pkg_dir.mkdir() + subprocess.run(["git", "init", "-q"], cwd=pkg_dir, check=True) + subprocess.run( + ["git", "remote", "add", "origin", "https://github.com/myorg/mypkg"], + cwd=pkg_dir, + check=True, + ) + populate_pkg_dir( + str(pkg_dir), + description="Test package", + root_url="https://github.com/myorg", + author="John Doe", + version="1.2.3", + verbose=False, + ) + return (pkg_dir / ".github" / "workflows" / "ci.yml").read_text() + + +def test_populate_writes_a_named_transport_stub(populated_ci): + assert "toJSON(secrets)" not in populated_ci + assert PYPI_LINE in populated_ci + assert _secrets_of(populated_ci) == { + "PYPI_PASSWORD": "${{ secrets.PYPI_PASSWORD }}" + } + assert "uses: i2mint/wads/.github/workflows/uv-ci.yml@master" in populated_ci + + +def _ci_file(tmp_path, text): + wf = tmp_path / ".github" / "workflows" + wf.mkdir(parents=True) + ci = wf / "ci.yml" + ci.write_text(text) + return ci + + +def test_rerendering_an_existing_json_stub_file_keeps_json(tmp_path): + """The Python API (used by fleet_migrate) must not flip an existing repo.""" + ci = _ci_file(tmp_path, STUB_TEMPLATE) + assert migrate_ci_to_stub(str(ci)) == STUB_TEMPLATE + + +def test_converting_an_inline_workflow_file_gives_named(tmp_path): + ci = _ci_file(tmp_path, "name: CI\non: [push]\njobs:\n t:\n runs-on: x\n") + assert "toJSON(secrets)" not in migrate_ci_to_stub(str(ci)) + + +def test_populate_warns_about_names_a_named_stub_cannot_pass(tmp_path, capsys): + """An existing pyproject declaring an out-of-superset secret gets the #63 warning.""" + pkg_dir = tmp_path / "mypkg" + pkg_dir.mkdir() + subprocess.run(["git", "init", "-q"], cwd=pkg_dir, check=True) + subprocess.run( + ["git", "remote", "add", "origin", "https://github.com/myorg/mypkg"], + cwd=pkg_dir, + check=True, + ) + (pkg_dir / "pyproject.toml").write_text( + '[project]\nname = "mypkg"\nversion = "0.1.0"\n' + '[tool.wads.ci.env]\nextra_envvars = ["COSMO_TEST_LEVEL"]\n' + ) + populate_pkg_dir( + str(pkg_dir), + description="Test package", + root_url="https://github.com/myorg", + author="John Doe", + version="1.2.3", + verbose=False, + ) + ci = (pkg_dir / ".github" / "workflows" / "ci.yml").read_text() + assert "COSMO_TEST_LEVEL: ${{ secrets.COSMO_TEST_LEVEL }}" in ci + assert "CANNOT START" in capsys.readouterr().err + + +def test_rerendering_json_stub_content_keeps_json(): + """``old_ci`` may be the workflow's content rather than a path.""" + assert migrate_ci_to_stub(STUB_TEMPLATE) == STUB_TEMPLATE + + +def test_auto_transport_falls_back_to_json_for_out_of_superset_secrets( + tmp_path, capsys +): + """Converting an EXISTING inline repo must never yield a stub that cannot start. + + A named stub passing a name outside the frozen superset fails at parse time + (issue #63). Without an explicit ``transport``, fall back to JSON, which + passes any name, and say so. + """ + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "demo"\nversion = "0.1.0"\n' + '[tool.wads.ci.env]\nrequired_envvars = ["MY_CUSTOM_TOKEN"]\n' + ) + ci = _ci_file(tmp_path, "name: CI\non: [push]\njobs:\n t:\n runs-on: x\n") + stub = migrate_ci_to_stub(str(ci)) + assert JSON_LINE in stub + assert "MY_CUSTOM_TOKEN" in capsys.readouterr().err + # An explicit choice is still honoured (with the loud #63 warning). + named = migrate_ci_to_stub(str(ci), transport="named") + assert "MY_CUSTOM_TOKEN: ${{ secrets.MY_CUSTOM_TOKEN }}" in named + + +def test_populate_leaves_a_custom_template_alone(tmp_path): + """Only the bundled stub is rewritten; a custom template is the user's.""" + custom = tmp_path / "my_ci.yml" + custom.write_text( + "name: CI\njobs:\n ci:\n secrets:\n" + render_stub_json_transport() + "\n" + ) + pkg_dir = tmp_path / "mypkg" + pkg_dir.mkdir() + subprocess.run(["git", "init", "-q"], cwd=pkg_dir, check=True) + subprocess.run( + ["git", "remote", "add", "origin", "https://github.com/myorg/mypkg"], + cwd=pkg_dir, + check=True, + ) + populate_pkg_dir( + str(pkg_dir), + description="Test package", + root_url="https://github.com/myorg", + author="John Doe", + version="1.2.3", + verbose=False, + ci_tpl_path=str(custom), + ) + ci = (pkg_dir / ".github" / "workflows" / "ci.yml").read_text() + assert ci == custom.read_text() + + +def test_the_cli_falls_back_to_json_for_out_of_superset_secrets(tmp_path): + """`wads-migrate ci-to-stub` on an inline workflow must not write an unstartable stub.""" + import sys + + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "demo"\nversion = "0.1.0"\n' + '[tool.wads.ci.env]\nrequired_envvars = ["MY_CUSTOM_TOKEN"]\n' + ) + from wads import github_ci_uv_path + + ci = _ci_file(tmp_path, open(github_ci_uv_path).read()) + out = tmp_path / "out.yml" + subprocess.run( + [sys.executable, "-m", "wads.migration", "ci-to-stub", str(ci), "-o", str(out)], + check=True, + capture_output=True, + text=True, + ) + assert JSON_LINE in out.read_text() + + +def test_on_demand_flip_of_an_inline_repo_never_writes_an_unstartable_stub(tmp_path): + from wads import github_ci_uv_path + from wads.ci_trigger import flip_to_on_demand + + (tmp_path / ".git").mkdir() + (tmp_path / "pyproject.toml").write_text( + '[project]\nname = "demo"\nversion = "0.1.0"\n' + '[tool.wads.ci.env]\nrequired_envvars = ["MY_CUSTOM_TOKEN"]\n' + ) + ci = _ci_file(tmp_path, open(github_ci_uv_path).read()) + flip_to_on_demand(tmp_path) + assert JSON_LINE in ci.read_text() diff --git a/wads/toml_util.py b/wads/toml_util.py index 289721b7..aca953cc 100644 --- a/wads/toml_util.py +++ b/wads/toml_util.py @@ -19,6 +19,48 @@ tomli_w = None +def pep639_license(license_name: str): + """The ``[project].license`` value for ``license_name``, PEP 639 when possible. + + A name that canonicalizes to a valid SPDX expression becomes that string + (``license = "MIT"``). Anything else keeps the deprecated + ``{"text": ...}`` table, because Hatchling rejects a ``license`` string + that is not valid SPDX -- so this never produces an unbuildable project. + Needs ``packaging>=24.2`` for SPDX support and falls back to the table + without it. + + >>> pep639_license("mit") + 'MIT' + >>> pep639_license("Apache Software License") + 'Apache-2.0' + >>> pep639_license("Proprietary") + {'text': 'Proprietary'} + """ + if isinstance(license_name, dict): + return license_name # already a table + if not isinstance(license_name, str): + return {"text": license_name} + try: + from packaging.licenses import ( + InvalidLicenseExpression, + canonicalize_license_expression, + ) + except ImportError: + return {"text": license_name} + aliases = { + "apache software license": "Apache-2.0", + "apache license 2.0": "Apache-2.0", + "apache 2.0": "Apache-2.0", + "mit license": "MIT", + "bsd license": "BSD-3-Clause", + } + candidate = aliases.get(str(license_name).strip().lower(), license_name) + try: + return canonicalize_license_expression(candidate) + except (InvalidLicenseExpression, TypeError): + return {"text": license_name} + + def read_pyproject_toml(pkg_dir: str) -> dict[str, Any]: """ Read pyproject.toml from the specified package directory.