v2: evidence track — benchmarks, stress, end-to-end storefront, docs

.github/workflows/ci.yaml

@@ -1,8 +1,9 @@
 name: CI
 
 on:
+  # Branches are tested through their pull request; a push trigger for them too ran every job twice.
   push:
-    branches: [main, "v2/**"]
+    branches: [main]
   pull_request:
 
 permissions:
@@ -22,6 +23,7 @@ jobs:
       - run: make lint
       - run: make types
       - run: make build
+      - run: make docs
 
   test:
     runs-on: ubuntu-latest
@@ -32,9 +34,15 @@ jobs:
         mongo: ["8.0"]
         experimental: [false]
         include:
+          - python: "3.11"
+            mongo: "7.0"
+            experimental: false
           - python: "3.13"
             mongo: "7.0"
             experimental: false
+          - python: "3.15"
+            mongo: "7.0"
+            experimental: false
           - python: "3.14t"
             mongo: "8.0"
             experimental: true
@@ -53,3 +61,16 @@ jobs:
         run: |
           docker compose -f docker/compose.rs.yaml logs --tail 100
           docker compose -f docker/compose.sharded.yaml logs --tail 100
+
+  tracing:
+    # End to end: Sentry SDK -> mongospecs-web -> MongoDB, received by a Sentry-compatible server.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: docker compose -f docker/compose.rs.yaml up -d --wait
+      - run: docker compose -f docker/compose.rustrak.yaml up -d
+      - run: uv sync --all-packages --python 3.13
+      - name: Wait for rustrak
+        run: timeout 60 sh -c 'until curl -sf http://localhost:8787/health >/dev/null; do sleep 1; done'
+      - run: uv run mongospecs-bench trace

.github/workflows/docs.yaml

@@ -0,0 +1,36 @@
+# Publishes the documentation site to GitHub Pages from main.
+# Requires Settings > Pages > Source: "GitHub Actions".
+name: Docs
+
+on:
+  push:
+    branches: [main]
+    paths: ["docs/**", "examples/**", "zensical.toml", ".github/workflows/docs.yaml"]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: true
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --group docs --python 3.13
+      - run: uv run --group docs zensical build --clean
+      - uses: actions/configure-pages@v5
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+      - id: deployment
+        uses: actions/deploy-pages@v4

.github/workflows/python-publish.yaml

@@ -11,7 +11,20 @@ permissions:
   contents: read
 
 jobs:
+  verify:
+    # Nothing is published from a commit that does not pass the checks.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: make up
+      - run: uv sync --all-packages --python 3.13
+      - run: make lint
+      - run: make types
+      - run: uv run pytest
+
   publish:
+    needs: verify
     runs-on: ubuntu-latest
     environment:
       name: pypi
@@ -29,5 +42,14 @@ jobs:
             mongospecs-v*) echo "package=mongospecs" >> "$GITHUB_OUTPUT" ;;
             *) echo "unrecognised release tag: $tag" >&2; exit 1 ;;
           esac
+      - name: The tag must be the version being published
+        run: |
+          package="${{ steps.pick.outputs.package }}"
+          declared="$(uv run --no-project python -c "import tomllib, sys; print(tomllib.load(open(sys.argv[1], 'rb'))['project']['version'])" "packages/$package/pyproject.toml")"
+          tagged="${GITHUB_REF_NAME#"$package"-v}"
+          if [ "$declared" != "$tagged" ]; then
+            echo "tag $GITHUB_REF_NAME says $tagged, but packages/$package/pyproject.toml says $declared" >&2
+            exit 1
+          fi
       - run: uv build --package ${{ steps.pick.outputs.package }}
       - uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -117,7 +117,7 @@ venv.bak/
 # Rope project settings
 .ropeproject
 
-# mkdocs documentation
+# zensical documentation
 /site
 
 # mypy

CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+Two packages are released from this repository; each entry says which one it is about. Versions
+follow semantic versioning, and while a package is below 1.0 a minor release may break
+compatibility. Breaking changes are marked **Breaking**.
+
+## Unreleased
+
+### mongospecs 0.2.0
+
+A rewrite. See [Migrating from 0.1.x](docs/migration.md) for the API map.
+
+- **Breaking:** async only, Python 3.11+, MongoDB 7.0+. The synchronous 0.1.x line stays on the
+  `0.1.x` branch and as `mongospecs<0.2`.
+- **Breaking:** models are bound to an explicit `Registry`; there is no class-level client.
+- **Breaking:** msgspec is no longer installed implicitly; choose an extra (`[msgspec]`,
+  `[pydantic]`, `[attrs]`).
+- Change tracking by snapshot: `save()` sends only what changed.
+- Typed, immutable queries with `Q`; `tracked`, `lean()` and `raw()` read modes; cursor and offset
+  pagination.
+- Atomic update operators, real upsert, `bulk_write`, `bulk_save`, partial-failure `insert_many`.
+- Relations: `Ref`, `Backref`, `include()`, cascade rules, `$graphLookup`, `$unionWith`.
+- Document hooks and plugins; timestamps, soft delete and revisions by field marker.
+- Transactions as a retried callback that restores in-memory documents when it aborts.
+- Index planning with gated destructive actions; change streams; per-command observability events
+  with logging, OpenTelemetry and Sentry listeners.
+- Document hooks run for `insert_many` and `bulk_save`.
+- `insert`, `save`, `insert_many` and `bulk_save` validate the document before writing it, on
+  every backend. `validate=False` on a call or `SpecConfig(validate_writes=False)` on a model
+  turns the check off.
+- A single-field update on a pydantic model runs the field's validators and not the model-level
+  ones, which could only see the other fields' defaults.
+
+### mongospecs-web 0.1.0
+
+First release.
+
+- Resources described as data and mounted on plain ASGI, Starlette, FastAPI, Litestar or Quart.
+- Field `Policy` drives the create, patch, put, read, filter and sort surface; guards and row
+  scope for authorization; RFC 9457 errors; ETag and `If-Match`; OpenAPI 3.1.
+- `PATCH` merges into embedded objects (RFC 7396); `Resource(merge_patch=False)` makes it replace
+  them. `PUT` is available as the `Put` operation.
+
+Changes made after the first review builds, for anyone who tried those:
+
+- **Breaking:** `?total=true` is opt-in per resource (`page=Cursor(total=True)`).
+- **Breaking:** `PATCH` on an embedded object is partial. Send `null` for keys to clear, or set
+  `merge_patch=False`.
+- **Breaking:** `If-Match` on a model without a `Revision` field is answered with 412.
+- **Breaking:** a filter or sort key given twice is a 400, as are `NaN` and `Infinity` in a body.
+- **Breaking:** a guard or hook registered for an operation the resource does not have raises
+  `ValueError`.

CONTRIBUTING.md

@@ -0,0 +1,51 @@
+# Contributing
+
+## Setting up
+
+You need [uv](https://docs.astral.sh/uv/) and Docker.
+
+```
+make install     # uv sync for the whole workspace
+make up          # MongoDB in Docker: a replica set on :27117 and a sharded cluster on :27118
+make check       # ruff, mypy --strict, and the whole test suite
+make down        # stop MongoDB and delete its data
+```
+
+Tests run against real MongoDB; nothing is mocked. `make test-fast` leaves out the tests that
+need the sharded cluster. To use another server, set `MONGOSPECS_TEST_URI` (a replica set) and
+`MONGOSPECS_TEST_SHARDED_URI`.
+
+## Where things are
+
+| Path | What |
+|---|---|
+| `packages/mongospecs` | the ODM |
+| `packages/mongospecs-web` | the REST layer and its framework plugs |
+| `bench` | benchmarks, stress scenarios and the storefront reference application; never published |
+| `examples` | the files shown in the tutorial; they are tests too |
+| `docs` | the documentation site (`make docs` builds it) |
+| `claudedocs/v2-plan.md` | the design of 0.2 and the record of every decision that departed from it |
+
+## What a change needs
+
+- **A test that fails without it.** For a bug, reproduce it first. Tests for behaviour that
+  differs by model library are parametrised over `msgspec`, `pydantic` and `attrs`.
+- **Docs for anything a user can see.** `python` blocks in `docs/guide` and `docs/web` are
+  executed in order, per page, against MongoDB; `py` blocks are shown but not run. Prefer `python`.
+- **A changelog entry** in `CHANGELOG.md`, marked **Breaking** when it is.
+- **`make check` passing.**
+
+Keep changes to one concern. If a change departs from the design in `claudedocs/v2-plan.md`, add
+an amendment row there saying what and why.
+
+## Plugs for other frameworks
+
+A plug only translates requests and responses. Prove it with the contract suite:
+`mongospecs_web.testing.run_contract` must produce the same transcript as the reference plug. See
+[Plugs](docs/web/plugs.md).
+
+## Releasing
+
+Each package is released by publishing a GitHub release whose tag names it:
+`mongospecs-v0.2.0` or `mongospecs-web-v0.1.0`. The tag must match the version in that package's
+`pyproject.toml`; the publish workflow checks this and runs the full suite before uploading.

Makefile

@@ -18,12 +18,12 @@ test-fast:
 	uv run pytest -m "not sharded"
 
 format:
-	uv run ruff format packages bench
-	uv run ruff check packages bench --fix
+	uv run ruff format packages bench examples
+	uv run ruff check packages bench examples --fix
 
 lint:
-	uv run ruff format --check packages bench
-	uv run ruff check packages bench
+	uv run ruff format --check packages bench examples
+	uv run ruff check packages bench examples
 
 types:
 	uv run mypy packages/mongospecs/src/mongospecs packages/mongospecs-web/src/mongospecs_web bench/src/mongospecs_bench
@@ -34,7 +34,7 @@ build:
 	uv build --all-packages
 
 docs:
-	uv run --group docs mkdocs build --strict
+	uv run --group docs zensical build --clean
 
 clean:
 	rm -rf dist build site .*_cache .venv-*

README.md

@@ -6,25 +6,44 @@ The aim is Mongoose's production feature set (change tracking, middleware, popul
 discriminators, transactions that restore document state) with typed Python models, plus
 [`mongospecs-web`](packages/mongospecs-web), a framework-neutral REST layer.
 
+**Documentation: <https://jaykv.github.io/mongospecs/>.** Start with
+[Getting started](https://jaykv.github.io/mongospecs/tutorial/).
+
 > **Status: 0.2.0, alpha.** This is a rewrite of 0.1.x: async-only, Python 3.11+, MongoDB 7.0+,
 > and not backwards compatible. The synchronous 0.1.x line lives on the
-> [`0.1.x` branch](https://github.com/jaykv/mongospecs/tree/0.1.x). The design and the work graph
-> are in [`claudedocs/v2-plan.md`](claudedocs/v2-plan.md).
+> [`0.1.x` branch](https://github.com/jaykv/mongospecs/tree/0.1.x). Changes are listed in the
+> [changelog](CHANGELOG.md).
 
 ## Install
 
 ```
 pip install "mongospecs[msgspec]"     # or [pydantic], [attrs], [all]
 ```
 
+You also need MongoDB 7.0 or newer. Transactions, cascading deletes and change streams need it to
+run as a replica set; a single node is enough. [`examples/compose.yaml`](examples/compose.yaml)
+starts one on `localhost:27017`:
+
+```
+docker compose -f examples/compose.yaml up -d --wait
+```
+
 ## A taste
 
+This is [`examples/quickstart.py`](examples/quickstart.py), as it is run by the test suite.
+
 ```python
+"""mongospecs in one file. Start MongoDB (see compose.yaml), then: python examples/quickstart.py"""
+
+import asyncio
+import os
 from datetime import datetime
 from typing import Annotated
 
 from mongospecs import CreatedAt, Inc, Index, Q, Ref, Registry, Revision, SpecConfig, UpdatedAt
-from mongospecs.msgspec import Spec          # or mongospecs.pydantic / mongospecs.attrs
+from mongospecs.msgspec import Spec  # or mongospecs.pydantic / mongospecs.attrs
+
+URI = os.environ.get("MONGO_URI", "mongodb://localhost:27017/quickstart?replicaSet=rs0")
 
 
 class User(Spec):
@@ -40,11 +59,11 @@ class Article(Spec):
     author: Ref[User] | None = None
     created_at: Annotated[datetime | None, CreatedAt] = None
     updated_at: Annotated[datetime | None, UpdatedAt] = None
-    rev: Annotated[int, Revision] = 0          # optimistic locking
+    rev: Annotated[int, Revision] = 0  # optimistic locking
 
 
-async def main():
-    async with Registry("mongodb://localhost/app", models=[User, Article]) as registry:
+async def main(uri: str = URI) -> None:
+    async with Registry(uri, models=[User, Article]) as registry:
         await registry.apply_indexes()
 
         ann = await User(name="ann").insert()
@@ -56,24 +75,34 @@ async def main():
         await article.save()
 
         # typed queries: names are checked, values are cast
-        popular = await Article.find(Q.views >= 100, Q.title.prefix("Hello")).sort("-views").limit(10)
+        popular = await Article.find(Q.views >= 0, Q.title.prefix("Hello")).sort("-views").limit(10)
+        assert [a.title for a in popular] == ["Hello, world"]
 
         # relations: one extra query, or a $lookup
         with_author = await Article.find(Q.author == ann).include("author").one()
-        print(with_author.author.doc.name)
+        print(with_author.title, "by", with_author.author.doc.name)
 
         # atomic updates, always validated; the updated document comes back
         article = await Article.update(article.id, Inc(views=1), tags=["intro"])
+        assert article.views == 1
 
-        # transactions: the callback is retried on transient errors, and in-memory
-        # documents are restored if it aborts
+        # transactions (these need a replica set): the callback is retried on transient
+        # errors, and in-memory documents are restored if it aborts
         async def publish(tx):
             article.tags.append("published")
             await article.save(session=tx)
 
         await registry.transaction(publish)
+        print("tags:", (await Article.get(article.id)).tags)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
 ```
 
+A REST API over a model is a few more lines; see [`examples/todo_api`](examples/todo_api) and the
+[web guide](https://jaykv.github.io/mongospecs/web/resources/).
+
 ## What is in the box
 
 | | |
@@ -87,13 +116,29 @@ async def main():
 | Operations | explicit `Registry` lifecycle, transactions, index planner with gated destructive actions, `$jsonSchema` validators, change streams, multi-database use, shard-key aware writes |
 | Observability | per-command events with model and operation names; logging, OpenTelemetry and Sentry listeners |
 
+## How it is tested
+
+- Unit and contract suites run against real MongoDB (a replica set and a sharded cluster), on
+  every backend, on Python 3.11 to 3.15.
+- A reference application, the storefront in `bench/`, is booted as a real server process and
+  driven over HTTP: a full customer journey on every plug, concurrent buyers that must never
+  oversell, multi-item orders that are all-or-nothing, a change-stream projector that must count
+  each sale exactly once across restarts, and orders that stay consistent across a primary
+  step-down.
+- Every example in the guide pages and in `examples/` is executed.
+- Benchmarks and stress scenarios in `bench/` run against real MongoDB; results and the
+  conclusions drawn from them are in `docs/benchmarks.md`.
+
 ## Development
 
 ```
 make install     # uv sync for the whole workspace
 make up          # MongoDB replica set + sharded cluster in Docker
-make check       # ruff, mypy --strict, tests against real MongoDB
+make check       # ruff, mypy --strict, tests against real MongoDB (unit, contract, end-to-end)
+uv run mongospecs-bench run          # benchmarks; `run LOAD` for the HTTP load test; `stress`; `trace`
 ```
 
 The repository is a uv workspace: `packages/mongospecs`, `packages/mongospecs-web` and `bench/`
-(benchmarks and stress scenarios against real MongoDB, never published).
+(benchmarks and stress scenarios against real MongoDB, never published). See
+[CONTRIBUTING.md](CONTRIBUTING.md); the design decisions behind 0.2 are recorded in
+[`claudedocs/v2-plan.md`](claudedocs/v2-plan.md).