Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 73 additions & 19 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,14 @@ on:
# workflow on the branch explicitly instead, so the fixed commit is checked.
workflow_dispatch:

# Pull requests run without production credentials. Forks receive only a
# read-only workflow token; neither collection nor test lanes may read secrets.
jobs:
check:
prepare:
runs-on: ubuntu-latest
outputs:
corrected: ${{ steps.autofix.outputs.changed }}
revision: ${{ steps.revision.outputs.sha }}
permissions:
contents: write
actions: write
Expand Down Expand Up @@ -40,9 +45,10 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Check out the branch itself (not the merge ref) so a fix can be
# committed back to it.
ref: ${{ github.head_ref || github.ref_name }}
# Test the event's exact revision; fixes push explicitly to its branch.
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- id: revision
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
Expand All @@ -68,6 +74,8 @@ jobs:
JOBTRACKER_SERVICE_TOKEN: ci
APP_ENCRYPTION_KEY: ci
GH_TOKEN: ${{ github.token }}
HEAD_BRANCH: ${{ github.head_ref }}
WORKFLOW_NAME: ${{ github.workflow }}
run: |
ruff check --fix src tests || true
ruff format src tests
Expand All @@ -91,8 +99,8 @@ jobs:
git config user.email "github-actions[bot]@users.noreply.github.com"
git add src tests openapi.json
git commit -m "chore: auto-fix lint, format and schema"
git push
gh workflow run "${{ github.workflow }}" --ref "${{ github.head_ref || github.ref_name }}"
git push origin "HEAD:$HEAD_BRANCH"
gh workflow run "$WORKFLOW_NAME" --ref "$HEAD_BRANCH"
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "::notice::Pushed mechanical fixes as a new commit; its own run judges the branch"
- name: Lint
Expand All @@ -110,26 +118,22 @@ jobs:
- name: Migrations apply cleanly
if: steps.autofix.outputs.changed != 'true'
run: python -m alembic upgrade head
# The WHOLE suite, with no marker filter. Three populations run here:
# the hermetic tests on a truncated database, the `corpus` tests against
# the generated catalog in tests/corpus.py - built from the committed
# measurement of production, so it needs no credential - and the
# `integration` tests, which skip because this database is not a copy of
# production and must never be one.
#
# NOTHING IN THIS FILE MAY READ A SECRET. That is the whole point: this
# workflow runs on pull_request, so anything it can reach is reachable
# from a branch an outside contributor can push. The workflow token is
# not a secret in that sense: on a fork's pull request it is read-only.
- name: Run tests
- name: Collect the complete test manifest
if: steps.autofix.outputs.changed != 'true'
env:
TEST_DATABASE_URL: postgresql://ci:ci@localhost:5432/jobtracker_ci
JOBTRACKER_SERVICE_TOKEN: ci
run: |
export TEST_REVISION=$(git rev-parse HEAD)
export APP_ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")
node --test tests/extension/*.test.cjs
pytest -q tests
python -m pytest -p tests.ci_suite --collect-only -q tests --test-report=manifest.json
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: steps.autofix.outputs.changed != 'true'
with:
name: test-manifest
path: manifest.json
if-no-files-found: error
- name: Models match migrations (autogen must be empty)
if: steps.autofix.outputs.changed != 'true'
run: |
Expand All @@ -138,3 +142,53 @@ jobs:
if grep -qE "op\.(add|drop|create|alter)" "$f"; then
echo "::error::ORM models drifted from migrations - generate a real migration"; cat "$f"; exit 1
fi

tests:
needs: prepare
if: needs.prepare.outputs.corrected != 'true'
strategy:
fail-fast: false
matrix:
lane: [shard-0, shard-1, shard-2, corpus]
uses: ./.github/workflows/test-run.yml
with:
lane: ${{ matrix.lane }}
shards: 3
revision: ${{ needs.prepare.outputs.revision }}

# Preserve the required check name. Every lane must pass and account for
# the complete collection before this gate can be green.
check:
if: always()
needs: [prepare, tests]
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Require successful preparation and execution
env:
PREPARE_RESULT: ${{ needs.prepare.result }}
TEST_RESULT: ${{ needs.tests.result }}
CORRECTED: ${{ needs.prepare.outputs.corrected }}
run: |
test "$PREPARE_RESULT" = success
if [ "$CORRECTED" != true ]; then
test "$TEST_RESULT" = success
fi
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
if: needs.prepare.outputs.corrected != 'true'
with:
ref: ${{ needs.prepare.outputs.revision }}
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
if: needs.prepare.outputs.corrected != 'true'
with:
name: test-manifest
path: collection
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
if: needs.prepare.outputs.corrected != 'true'
with:
pattern: timings-*
path: measurements
- name: Verify every collected test executed exactly once
if: needs.prepare.outputs.corrected != 'true'
run: python tools/test_performance.py measurements --shards=3 --manifest=collection/manifest.json | tee "$GITHUB_STEP_SUMMARY"
39 changes: 39 additions & 0 deletions .github/workflows/test-performance.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Test performance

on:
workflow_dispatch:

permissions:
contents: read

jobs:
benchmark:
strategy:
fail-fast: false
matrix:
lane: [serial, profile, shard-0, shard-1, shard-2, corpus]
repetition: [1, 2]
exclude:
- lane: profile
repetition: 2
uses: ./.github/workflows/test-run.yml
with:
lane: ${{ matrix.lane }}
repetition: ${{ matrix.repetition }}
shards: 3
revision: ${{ github.event.pull_request.head.sha || github.sha }}

compare:
if: always()
needs: benchmark
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
pattern: timings-*
path: measurements
- name: Verify coverage and compare timings
run: python tools/test_performance.py measurements --repetitions=2 --shards=3 | tee "$GITHUB_STEP_SUMMARY"
79 changes: 79 additions & 0 deletions .github/workflows/test-run.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Python test lane

on:
workflow_call:
inputs:
lane:
type: string
required: true
revision:
type: string
required: true
shards:
type: number
required: true
repetition:
type: number
default: 1

permissions:
contents: read

jobs:
run:
name: ${{ inputs.lane }}
runs-on: ubuntu-latest
timeout-minutes: 15
services:
postgres:
image: pgvector/pgvector:pg18-trixie@sha256:78bf48b801e792f99e3ac62b5036fd3876e9be48afda16c1e331af1c75ceb2ff
env:
POSTGRES_USER: ci
POSTGRES_PASSWORD: ci
POSTGRES_DB: jobtracker_ci
ports: ['5432:5432']
options: >-
--health-cmd "pg_isready -U ci" --health-interval 5s
--health-timeout 5s --health-retries 10
env:
DATABASE_URL: postgresql://ci:ci@localhost:5432/jobtracker_ci
TEST_DATABASE_URL: postgresql://ci:ci@localhost:5432/jobtracker_ci
JOBTRACKER_SERVICE_TOKEN: ci
PYTHONPATH: src
TEST_SHARDS: ${{ inputs.shards }}
TEST_LANE: ${{ inputs.lane }}
TEST_REPETITION: ${{ inputs.repetition }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ inputs.revision }}
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
- run: uv sync --frozen
- run: echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
- run: python -m alembic upgrade head
- name: Measure tests
run: |
mkdir -p results
export TEST_REVISION=$(git rev-parse HEAD)
export APP_ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")
args=(-p tests.ci_suite -q tests --durations=25 --test-report=results/timing.json --junitxml=results/junit.xml)
case "$TEST_LANE" in
serial|profile) ;;
corpus) args+=(-m corpus) ;;
shard-*) args+=(-m 'not corpus' --shard-count="$TEST_SHARDS" --shard-index="${TEST_LANE#shard-}") ;;
*) echo "Unknown test lane: $TEST_LANE"; exit 1 ;;
esac
if [ "$TEST_LANE" = profile ]; then
python -m cProfile -o results/profile.pstats -m pytest "${args[@]}"
else
python -m pytest "${args[@]}"
fi
- name: Upload measurements
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: timings-${{ inputs.lane }}-${{ inputs.repetition }}
path: results/
if-no-files-found: error
83 changes: 83 additions & 0 deletions docs/testing-performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Test performance

Measured on GitHub-hosted Ubuntu runners on 2026-09-09, revision
`3271201a9fce2757a093d8da2e199e8a17284f97`.
[Experiment and artifacts](https://github.com/kensac/job-scripts/actions/runs/34328307413).

## Measured comparison

Each parallel trial used three deterministic shards of tests without the
`corpus` marker and one corpus job, with an independent PostgreSQL database
per job. Both trials matched the serial run's 1,471 test IDs and outcomes
exactly: 1,457 passed and 14 skipped. Subtests remain inside their parent case.

| Trial | Serial pytest | Parallel pytest critical path | Serial job | Parallel job span | Parallel runner total |
|---|---:|---:|---:|---:|---:|
| 1 | 171.52s | 96.47s | 201s | 135s | 350s |
| 2 | 185.86s | 57.90s | 214s | 90s | 307s |

Job times include provisioning, dependency installation and artifact upload.
Parallel span is the earliest parallel job start through the last job finish;
it is not the entire CI workflow, which also performs static checks and a
final coverage check. Queue delays before jobs start are excluded. Two trials
support adopting this split, but do not establish a stable speedup: one shard
was considerably slower in trial 1, and the cause was not determined.
Parallel execution consumes more total runner time.

## Where time goes

In ordinary, unprofiled serial execution, setup phases consumed 132.81s and
145.02s; test call phases consumed 31.18s and 33.17s. Teardown was about 0.32s.
Four corpus setup phases each took about six seconds. Grouping corpus tests
into their own job also avoids rebuilding it between ordinary tests.

A separate cProfile run counted four corpus builds and 1,433 `_reseed` calls.
The current `_reseed` implementation inserts two group defaults and 24
application defaults individually, so those calls account for 37,258 default
insert executions. Profiled cumulative times overlap and instrumentation
substantially increases runtime; they must not be used as ordinary wall times.

The worker entrypoint test's call took about 8.02 seconds in both serial runs.
Its source waits a fixed eight seconds to detect process fallthrough.

## Adopted design and next experiments

Normal CI shares its lane implementation with the manual benchmark. A full
collection manifest and the lane reports must identify the same commit and
account for every case exactly once before the required `check` passes.
Each lane retains the existing database isolation and cleanup fixtures.
Lint, formatting, type checks, compilation, migration checks and extension
tests remain required. Mechanical lint and formatting corrections still run
before the tests.

Prioritize subsequent work in this order:

1. Batch default reseeding and measure setup time again. Preserve fresh state
per test, including defaults and sequences; transaction rollback is not a
replacement for tests that commit or use separate connections.
2. Introduce an explicit database-free test population after auditing actual
imports and fixture dependencies. A missing `db` fixture argument alone
cannot establish that a test does not access the database.
3. Replace the worker test's fixed grace period with observable worker-loop
progress and a bounded failure timeout. Process liveness alone does not
prove it reached the loop.
4. Revisit shard balancing only after repeated timing reports show a persistent
imbalance. Keep deterministic assignment and exact coverage verification.

These follow-up changes are recommendations, not measured implementations.

## Reproduce

Dispatch the `Test performance` workflow on a selected revision. It runs two
serial and parallel trials plus one separate call profile. Download its
`timings-*` artifacts, preserving their directories, then run:

```sh
python tools/test_performance.py measurements --repetitions=2 --shards=3
```

The comparison rejects missing, duplicated or changed test outcomes and
reports from different revisions. Compare job timestamps as well as pytest
phase times before changing job counts. Normal CI also uploads these phase
reports, so later changes can be evaluated without adding a profiler to the
regular test path.
Loading
Loading