Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,5 @@ and direct reviewers to the riskiest parts. Write "None" when there are none.
behavior.
- [ ] I confirmed that each changed plugin remains self-contained, or this
change does not affect plugin content.
- [ ] I ran `bun run check:python`, or this change does not affect registered
Python packages or their repository quality infrastructure.
78 changes: 78 additions & 0 deletions .github/workflows/python-quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: Python quality

on:
pull_request:
push:
branches: [main]
workflow_dispatch:

permissions:
contents: read

jobs:
package:
name: Python ${{ matrix.python-version }} on ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.10", "3.11", "3.12", "3.13"]
runs-on: ${{ matrix.os }}
env:
UV_PYTHON: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- run: scripts/check-python

inventory:
name: Python inventory guard
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: scripts/test-python-inventory

fresh-install:
name: Python fresh plugin install
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- uses: astral-sh/setup-uv@v6
- run: bash plugins/capability/darrow-observability-langfuse/tests/fresh-install.test.sh

performance:
name: Python performance invariants
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- uses: astral-sh/setup-uv@v6
- run: plugins/capability/darrow-observability-langfuse/tests/performance.test.sh

aggregate:
name: Python quality
if: always()
needs: [package, inventory, fresh-install, performance]
runs-on: ubuntu-latest
steps:
- name: Require every Python quality job
env:
PACKAGE_RESULT: ${{ needs.package.result }}
INVENTORY_RESULT: ${{ needs.inventory.result }}
INSTALL_RESULT: ${{ needs.fresh-install.result }}
PERFORMANCE_RESULT: ${{ needs.performance.result }}
run: |
test "$PACKAGE_RESULT" = success
test "$INVENTORY_RESULT" = success
test "$INSTALL_RESULT" = success
test "$PERFORMANCE_RESULT" = success
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ evals/results/
evals/corpus/orchestration/cache/
.tmp/
.DS_Store
**/.venv/
**/__pycache__/
**/.pytest_cache/
**/.hypothesis/
**/.coverage
**/coverage.json

# Share Codex configuration without tracking other local Codex state.
!.codex/
Expand Down
1 change: 1 addition & 0 deletions .lintstagedrc.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"*": "prettier --check --ignore-unknown --no-error-on-unmatched-pattern",
"*.ts": "eslint --no-warn-ignored --max-warnings 0",
"**/{*.py,pyproject.toml,uv.lock}": "scripts/check-python --fast",
"*.{sh,bash}": "scripts/lint-shell",
".githooks/!(*.*)": "scripts/lint-shell",
"scripts/!(*.*)": "scripts/lint-shell",
Expand Down
4 changes: 4 additions & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
plugins/*/*/skills/*/SKILL.md
**/.venv/**
**/__pycache__/**
**/.pytest_cache/**
**/.hypothesis/**
**/.coverage
**/coverage.json
.worktrees/
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ Review agents must not run Git or GitHub commands against this repository.
Before creating, changing, or running evals,
read [`docs/eval-development.md`](docs/eval-development.md).

All Python packages follow [`docs/specs/python-quality.md`](docs/specs/python-quality.md).
Run `bun run check:python` after changing Python source, tests, project metadata,
or locks.

## Git

- Use imperative Conventional Commits without trailing periods.
Expand Down
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,15 @@ bun run lint:shell
bun run typecheck
bun run check:decisions
bun run check:docs
bun run check:python
```

`check:decisions` rejects missing or stale checked-in ADR catalog data. When
ADRs change, refresh it first with
`bash plugins/foundation/darrow-decisions/bin/decision catalog rebuild --repo .`.
`check:python` is required when Python source, tests, package metadata, or locks
change; its standards and package inventory are defined in
[`docs/specs/python-quality.md`](docs/specs/python-quality.md).

Run script tests with both `bash` and `/bin/bash`. Run relevant evals with:

Expand Down
1 change: 1 addition & 0 deletions docs/specs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ These files define current invariants. Start with the
- [Code review](code-review.md): independent review and repair verification.
- [Verification](verification.md): replaceable assessment and criterion-level evidence.
- [Langfuse observability](observability-langfuse.md): Codex telemetry and privacy.
- [Repository Python quality](python-quality.md): package inventory, checks, tests, and release evidence.
- [Marketplace installation](marketplace-installation.md): inventory-driven host installation shortcuts.
- [Adaptive Delivery](adaptive-delivery.md): one explicit execution owner.
- [Ticket to PR](ticket-to-pr.md): the delivery recipe.
Expand Down
15 changes: 14 additions & 1 deletion docs/specs/observability-langfuse.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,16 @@ continuity, missing-snapshot quarantine, epoch session segmentation, detached
and non-ticket Git state, rollout reconstruction, deduplication, content
privacy, malformed input, missing runtime or configuration, and exporter
failure. Backend checks run through UV. Portable hook-launcher tests run with
both supported Bash executables.
both supported Bash executables. The backend conforms to the repository-wide
[Python quality standard](python-quality.md), including separate 95% statement
and branch coverage gates on every supported Python and CI platform.

Performance evidence distinguishes startup from steady-state foreground
capture, confirms network-independent foreground completion, bounded
incremental reads and exporter batching, and reports peak memory for a stated
workload. Release evidence includes a fresh artifact install with runtime-only
locked dependencies. The isolated live Langfuse smoke test remains the release
check for actual ingestion.

Participant-visible colocated evals cover configuration/help intent, refusal to
claim tracing without prerequisites, privacy disclosure, work-item precedence,
Expand Down Expand Up @@ -314,3 +323,7 @@ trace and work-item metadata.
8. **OLF-P8 — Real ingestion.** When Docker infrastructure is available, an
isolated official Langfuse Compose deployment accepts and exposes a trace
emitted by the plugin.
9. **OLF-P9 — Python quality.** The backend passes the registered-package
formatting, lint, strict typing, deterministic tests, separate statement
and branch coverage, supported-platform CI, performance, and fresh-install
obligations in the repository Python quality standard.
82 changes: 82 additions & 0 deletions docs/specs/python-quality.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Repository Python quality

## Purpose

Python in this repository must remain readable, typed, deterministic, and safe
to release across its declared support range. These rules apply to every Python
package listed in [`python-packages.txt`](../../python-packages.txt). A new
Python source file, `pyproject.toml`, or UV lock outside a registered package is
an error until its package is added to that inventory.

## Source quality

Packages use Ruff for formatting, import order, common correctness rules,
modern syntax, naming, simplification, and a cyclomatic-complexity ceiling of
five. Mypy runs in strict mode over production code, tests, and package tooling.
Development tools belong in a locked development dependency group and must not
enter the runtime dependency set.

Names describe the domain concept or action they represent. A module owns one
coherent responsibility; a function performs one focused operation at one
level of abstraction. Boundary code validates untrusted input and reports a
specific error. An exception may be translated when the new error adds useful
domain context, but it must retain its cause. Broad catches are allowed only at
an explicit process or protocol boundary with documented failure semantics.

Automation cannot judge every aspect of those rules. Reviewers inspect names,
module cohesion, function focus, error boundaries, and whether assertions
verify observable behavior rather than implementation details.

## Tests and coverage

Each package keeps fast unit tests for local behavior and integration tests for
the real filesystem, database, subprocess, or protocol seams it owns. Every bug
fix carries a regression test that fails for the reported trigger. Property
tests are required where generated cases can test a meaningful invariant, such
as parser normalization, validation, redaction, serialization round trips, or
state-machine transitions. They use deterministic settings suitable for CI.

Coverage is measured for the complete import package with branch measurement
enabled. The aggregate gate calculates statement coverage and branch coverage
separately for each registered package; both must be at least 95%. The gate
must fail when statement coverage passes but branch coverage does not. Coverage
exclusions require a concrete unreachable-platform or interpreter reason in
the source and review.

Each package owns its coverage source and branch configuration. The repository
quality infrastructure owns the shared percentage calculation, threshold, and
failure behavior so packages do not copy or weaken that policy.

## Commands and enforcement

`bun run check:python` is the canonical local and CI command. For every
registered package it verifies the lock, synchronizes the complete development
environment, checks formatting and lint, runs strict typing, executes all tests
including property tests, and enforces both coverage measures. The pre-commit
hook runs the fast formatting, lint, and type portion when staged Python or
Python project files change.

CI runs the canonical gate on Linux and macOS for every declared Python minor
version. A stable aggregate `Python quality` status is the branch-protection
check. The inventory guard itself is tested by introducing a temporary
unregistered package and proving that the gate refuses it.

## Performance and release evidence

Performance checks target observable workload invariants rather than arbitrary
timing budgets. A benchmark identifies its fixture size and environment,
reports startup and steady-state foreground capture separately, confirms that
foreground capture does not depend on the network, checks bounded incremental
rollout reads and exporter batching, and reports peak memory. A numeric
regression threshold may be added only after representative measurements make
the budget defensible.

Plugin-specific benchmark workloads, comparison rules, and executable entrypoints
live inside the plugin that owns the behavior. Repository commands and CI may
invoke those entrypoints, but do not reproduce the plugin's internal path
resolution or benchmark logic.

Release validation installs the plugin from a fresh copied artifact using only
locked runtime dependencies before exercising its launcher and backend. The
isolated live Langfuse ingestion procedure remains release evidence for the
external service boundary; it is not replaced by mocked CI tests.
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
"check:docs": "bun scripts/check-docs.ts",
"check:docs:external": "bun scripts/check-docs.ts --external",
"check:decisions": "bash plugins/foundation/darrow-decisions/bin/decision validate --repo .",
"check:python": "scripts/check-python",
"check:python:inventory": "scripts/test-python-inventory",
"check:python:performance": "plugins/capability/darrow-observability-langfuse/tests/performance.test.sh",
"eval": "bun evals/runner/run.ts",
"eval:coverage": "bun evals/runner/coverage.ts",
"eval:orchestration:prepare": "bun evals/corpus/orchestration/prepare.ts",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "darrow-observability-langfuse",
"description": "Export Codex rollout turns to Langfuse with attribution epochs",
"version": "0.4.2",
"version": "0.5.0",
"license": "BUSL-1.1",
"author": {
"name": "Björn Rochel",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "darrow-observability-langfuse",
"version": "0.4.2",
"version": "0.5.0",
"description": "Export Codex rollout turns to Langfuse with attribution epochs",
"author": {
"name": "Björn Rochel",
Expand Down
4 changes: 3 additions & 1 deletion plugins/capability/darrow-observability-langfuse/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,10 +264,12 @@ bash hooks/stop.test.sh
bash hooks/export-failure.test.sh
bash hooks/refusal.test.sh
bash hooks/strict.test.sh
uv run --frozen --project backend python -m unittest discover -s backend/tests
bun run check:python
```

Run each shell test with both `bash` and `/bin/bash` in repository development.
The repository command verifies the UV lock, formatting, lint, strict typing,
tests, property tests, and separate statement and branch coverage gates.

## When to use

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,51 @@ dependencies = [
"opentelemetry-exporter-otlp-proto-http>=1.36,<2",
]

[dependency-groups]
dev = [
"coverage[toml]>=7.10,<8",
"hypothesis>=6.138,<7",
"mypy>=1.18,<2",
"pytest>=8.4,<9",
"ruff>=0.12,<1",
"types-protobuf>=6.32,<7",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv]
default-groups = []

[tool.hatch.build.targets.wheel]
packages = ["src/darrow_observability_langfuse"]

[tool.ruff]
target-version = "py310"
line-length = 88
src = ["src", "tests", "scripts"]

[tool.ruff.lint]
select = ["B", "C4", "C90", "E4", "E7", "E9", "F", "I", "N", "RUF", "SIM", "UP"]
mccabe.max-complexity = 5

[tool.ruff.format]
docstring-code-format = true

[tool.mypy]
python_version = "3.10"
strict = true
files = ["src", "tests", "scripts"]

[tool.pytest.ini_options]
addopts = ["--strict-config", "--strict-markers"]
testpaths = ["tests"]

[tool.coverage.run]
branch = true
source = ["darrow_observability_langfuse"]

[tool.coverage.report]
show_missing = true
skip_covered = false
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Development-only quality gate helpers."""
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any


def _load(path: Path) -> dict[str, Any]:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise ValueError(f"benchmark output is not an object: {path}")
return value


def verify(control: dict[str, Any], candidate: dict[str, Any]) -> None:
matched = ("turns", "input_sha256", "observations", "acknowledged_turns")
for key in matched:
if control.get(key) != candidate.get(key):
raise ValueError(f"benchmark workloads differ for {key}")
if candidate["bytes_read"] >= control["bytes_read"]:
raise ValueError("candidate incremental reads are not bounded below control")
if candidate["exporter_requests"] > control["exporter_requests"]:
raise ValueError("candidate exporter batching regressed")


def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("control", type=Path)
parser.add_argument("candidate", type=Path)
arguments = parser.parse_args()
control = _load(arguments.control)
candidate = _load(arguments.candidate)
verify(control, candidate)
print(json.dumps({"control": control, "candidate": candidate}, sort_keys=True))


if __name__ == "__main__":
main()
Loading
Loading