🧺 Sift — evidence-based PR triage for open-source maintainers

Open-source maintainers are drowning in low-quality, AI-generated pull requests. Creating a PR now costs seconds; reviewing one still costs minutes. Sift screens every incoming PR and hands the maintainer a scorecard backed by quoted evidence — so "read 80 lines of slop" becomes "glance at 3 lines of proof".

Non-negotiable principles

1. Diagnostic only. Sift never auto-closes, never edits code, never merges. A human always decides.
2. Evidence, not vibes. Every finding quotes the offending lines, numbers, or timestamps. No black-box scores.
3. Not an AI detector. We flag symptoms of low-effort contributions (placeholder tests, machine-speed commits, empty descriptions) — bad signals are bad no matter who wrote them. Sift never claims to detect "AI authorship".
4. Local & self-sufficient. Zero LLM, zero third-party APIs in the core engine. Deterministic, cheap, fast.
5. Kind to contributors. Low scores come with actionable reasons, so a good-faith author can fix and resubmit.

Checks (v0.1 — all deterministic, Layer 0 + Layer 1)

Check	Layer	What it catches
no-description / thin-description	mechanical	empty or one-line PR body
diff-desc-mismatch	mechanical	+150 lines with <80 chars of explanation
placeholder-tests	mechanical	assert True, expect(true).toBe(true), t.Skip, empty test bodies
overcommented	mechanical	≥45% comment lines or boilerplate clichés ("Here's the implementation…")
todo-flood	mechanical	≥5 new TODO/FIXME/HACK markers
no-tests	mechanical	≥50 lines of new source logic, zero test files touched
commit-velocity	behavioral	5+ commits within 120s; big PR authored in <60s
shotgun	behavioral	author opened 5+/10+ PRs across GitHub in 24h
ghost	behavioral	account created ≤3 days before the PR
commit-bomb	behavioral	≥500 additions in a single commit

Quick start

pip install .

# Offline demo (fixture = normalized PR data)
sift scan --fixture tests/fixtures/slop_pr.json
sift scan --fixture tests/fixtures/clean_pr.json

# Live scan of a real PR (read-only API calls)
export GITHUB_TOKEN=***
sift scan --pr owner/repo#123

Exit code: 0 when score ≥ --fail-below (default 40), else 1. Useful for CI gating — but gating is the maintainer's choice, not ours.

GitHub Actions

Copy .github/workflows/sift.yml.example into the target repo. Notes:

• Runs on pull_request (never pull_request_target): fork code gets a read-only token, no secrets. Sift reads metadata only anyway.
• Posts the scorecard as a PR comment and uploads a JSON artifact.
• persist-credentials: false on checkout, minimal permissions: block.

Score bands

Score	Band	Suggested posture
85–100	✅ Healthy	review as usual
60–84	👀 Normal	minor smells listed
40–59	⚠️ Needs attention	read evidence before spending time
0–39	🚧 Quarantine suggested	maintainer decides

Roadmap

• Layer 2 — sandbox execution is available through the unprivileged executor and the customized Git builtin's --exec mode. Automated issue reproduction and stronger self-hosted isolation remain future work.
• Layer 3 — optional evidence-constrained review is available in the customized Git builtin through opt-in OpenRouter (--llm) or fully offline replay (--llm-response). The Python API can request the opt-in live mode.
• GitHub App shell (Probot): Checks API integration, maintainer queue, /sift approve|reject commands.
• Prompt-injection hardening for any layer that ingests untrusted text.

Development

pip install -e ".[dev]" 2>/dev/null || pip install -e . pytest
pytest

License: MIT