Establish and enforce repository-wide Python quality standards #172

Outcome

Establish and enforce one Python quality standard for the entire existing Python baseline and all future Python contributions, whether written by people or generated. Bring the current Langfuse backend into compliance and make the checks reproducible locally and mandatory in CI.

The agreed bar is strict typing, cyclomatic complexity at most 5 per function, at least 95% line coverage and 95% branch coverage separately per package, and property tests wherever meaningful invariants exist. Dedicated performance checks apply to the observability plugin.

Context

The current Python implementation lives in plugins/capability/darrow-observability-langfuse/backend/. It already has unittest tests and a capture benchmark, but its project configuration has no Python lint or type-checking gates. The checked-in GitHub Actions workflow covers documentation and TypeScript checks, and the active main-protection ruleset currently has no required status checks.

Done criteria

• Document the canonical Python standard under docs/specs/, with concise references from AGENTS.md, CONTRIBUTING.md, and the PR template. Require intent-revealing names, cohesive modules, focused functions, and explicit error handling. Review naming, responsibility boundaries, and assertion quality alongside automated metrics.
• Configure Ruff formatting and linting, including a maximum cyclomatic complexity of 5 per function, and strict mypy checks. Cover maintained Python implementation, tests, and scripts. Validate external data at runtime and keep type escape hatches narrowly justified at dependency boundaries. Do not weaken checks or add blanket suppressions merely to obtain passing results.
• Manage and lock the development toolchain through uv: Ruff, mypy, pytest, Hypothesis, and Coverage.py. Validate lockfile freshness in CI, keep development tools out of plugin runtime installs, and preserve each plugin's independent installation and self-contained dependencies/configuration.
• Provide one documented repository command that runs the applicable Python quality checks. Ensure new Python projects or source files cannot silently fall outside the checked inventory. Add fast Python checks to the existing local hook workflow.
• Bring all existing Python code into compliance, preserving public behavior, durable-state compatibility, and privacy guarantees. Add meaningful tests before behavior-preserving refactoring and retain useful existing unittest tests, which pytest can run.
• Require unit tests for observable behavior, edge cases, and failure paths; add regression tests for bug fixes. Complement unit tests with deterministic integration tests for relevant filesystem, SQLite, subprocess, and CLI boundaries.
• Enforce at least 95% line coverage and at least 95% branch coverage independently for each implementation package, including source files never imported by tests. Keep uncovered code visible and explained during review. Verify the gate rejects a case that passes line coverage but fails branch coverage; a combined coverage percentage is insufficient.
• Require Hypothesis property tests wherever meaningful invariants exist. Apply this to suitable parser, normalization, redaction, and state-transition behavior. Test public contracts rather than mirroring implementation details; property tests complement example-based unit tests.
• Run formatting, linting, typing, deterministic tests, property tests, and coverage in GitHub Actions across declared Python/platform support. Add the aggregate Python quality result as a required check in the existing repository ruleset so failing checks block merging.
• Add release validation from a fresh installation of the deliverable plugin, including its launchers and runtime dependencies. Retain isolated Langfuse ingestion testing as release evidence.
• Scope dedicated performance verification to observability. Cover the existing incremental-read, network-independent foreground capture, bounded batching, and terminal-hook timing requirements in docs/specs/observability-langfuse.md. Measure capture latency and memory, distinguish startup from steady-state capture, and avoid inventing additional timing thresholds without representative measurements. Other Python code should use sensible algorithms and bounded resource use without a mandatory benchmark suite.

Scope boundary

Broader Bash-to-Python migration and native Windows enablement remain part of the parent research and subsequent migration work. This task establishes the quality baseline without changing the current runtime-adoption decisions or promising Windows support for the existing POSIX-dependent backend.