Agent Reviewer

Evidence-backed candidate repository evaluation service for local operators.

Agent Reviewer takes a candidate's project repository (plus optional resume and consented GitHub identity), runs profile-specific automated checks inside constrained Docker workers, and returns a blind, evidence-backed, advisory result. Candidate identity is never fed into scoring, every finding is traced back to a source file, and a human always records the final decision.

This milestone is a production-oriented vertical slice, not a public SaaS release. It is designed to:

• ingest a candidate repository URL and optional resume PDF
• optionally enrich a consented GitHub identity
• run profile-specific automated checks inside constrained Docker workers
• collect blind review inputs for the evaluator
• persist attempts, retries, checkpoints, artifacts, and operator decisions
• expose a minimal FastAPI web UI for submission and result review

Product tour

The screenshots below are captured from the reproducible browser fixture (agent_reviewer.browser_fixture) — the same deterministic build the Playwright suite drives — so what you see here is exactly what the shipped UI renders.

1. Submit an evaluation

Operators describe the role, point at a public Git URL, and pick an evaluation profile (or let it auto-detect the stack). Resume and GitHub enrichment are explicitly optional — the form and sidebar make the blind-review guarantees first-class: candidate identity is excluded from scoring, enrichment is advisory evidence only, and every result requires human review. Reusable templates pre-fill the role, prompt, and profile in one click, and a live "recent evaluations" rail shows recently completed scores.

2. Read an evidence-backed result

Each result leads with a score gauge and fit verdict, then qualifies it with signal badges — High confidence, Blind review passed, Resume evidence, GitHub evidence — and per-category score bars. Findings are split into strengths and risks, and the Evidence panel cites the exact source location (…/Controllers/HomeController.php:1) behind the reviewer's claims. When more than one candidate has completed for the same role, a Candidate ranking block compares them side by side. Nothing is a black box: a banner states plainly that the AI is decision support only, and the Human decision panel forces a person to record Advance / Hold / Decline with a written rationale for the audit trail. A data-and-privacy block shows the automatic PII erasure date and a one-click purge.

3. Score a whole pipeline at once

Upload a candidate CSV and every valid repository is queued as one batch. The batch view is an operator cockpit: import/queue summary tiles, per-row import status (Created / Reused / Skipped) with a Pending-delivery badge, full-text search, status filters, and a direct link from each row into its individual evaluation. As with single submissions, candidate names and emails stay in this internal operator view only. A Redrive action re-queues any candidate whose delivery has not been confirmed.

4. Standardize with reusable templates

Templates capture a role's job description, task requirements, evaluation prompt, and profile as a reusable preset so every candidate for the same role is judged against identical criteria. Built-in templates ship read-only; operators can author their own and optionally lock them.

Security warnings

• The web UI currently requires HTTP Basic authentication. The temporary internal-only credential is alpetg:alpetg; replace it with a real identity provider before any wider deployment.
• Candidate repositories are hostile input. Run Docker workers on dedicated hosts with disposable workspaces.
• Dependency installation and repository acquisition require outbound network access unless you pre-seed artifacts.
• SQLite is for local and single-node operation only. Use PostgreSQL plus Redis/RQ for multi-worker durability.
• No provider fallback fabricates success. Missing model credentials should end in authentication_unavailable.

Prerequisites

• Python 3.12
• uv
• Docker Engine
• Node 18+ (npm)
• Chromium install support for Playwright

Required for live OpenHands review:

• a valid OpenHands/OpenAI Codex subscription login already configured on the host. This is the default no-key path.
• LLM_API_KEY remains optional for a later API-key deployment.

Environment

Copy .env.example to .env and set the fields you actually need.

Important fields:

• APP_ENV=development|test|production
• QUEUE_BACKEND=eager|rq
• EVALUATOR_BACKEND=openhands|fixture
• DATABASE_URL
• REDIS_URL
• LLM_PROVIDER
• LLM_MODEL
• LLM_API_KEY (optional when using the OpenHands Codex subscription)
• GITHUB_TOKEN

Local setup

Install Python dependencies:

uv sync --dev

Install browser-test dependencies:

npm ci
npx playwright install chromium

Build Docker images used by the evaluator:

docker build -f docker/runner/Dockerfile.laravel -t agent-reviewer/laravel-runner:1.0.0 .
docker build -f docker/runner/Dockerfile.react-native -t agent-reviewer/react-native-runner:1.0.0 .
docker build -f docker/ingest/Dockerfile.pdf -t agent-reviewer/pdf-ingest:1.0.0 docker/ingest

Running locally

Eager local mode

This is the fastest loop for UI and schema work.

APP_ENV=development \
QUEUE_BACKEND=eager \
EVALUATOR_BACKEND=openhands \
uv run python -m agent_reviewer.web

Startup-only validation:

uv run python -m agent_reviewer.web --check-startup

Deterministic UI fixture

To browse the UI with seeded, deterministic data (no model calls, no Docker) — the exact build the screenshots above were captured from:

env APP_ENV=test \
QUEUE_BACKEND=eager \
EVALUATOR_BACKEND=fixture \
GITHUB_ENRICHMENT_ENABLED=true \
LLM_API_KEY=test-key \
APP_DATA_DIR=.playwright-data \
ARTIFACT_ROOT=.playwright-data/artifacts \
uv run python -m agent_reviewer.browser_fixture --host 127.0.0.1 --port 8000

Then open http://127.0.0.1:8000/ and sign in with the temporary Basic credential alpetg:alpetg.

Durable queue mode

Start PostgreSQL and Redis:

docker compose up -d postgres redis

Run the web process:

APP_ENV=development \
QUEUE_BACKEND=rq \
DATABASE_URL=postgresql+psycopg://agent_reviewer:agent_reviewer_local@127.0.0.1:5432/agent_reviewer \
REDIS_URL=redis://127.0.0.1:6379/0 \
uv run python -m agent_reviewer.web

Run the worker:

APP_ENV=development \
QUEUE_BACKEND=rq \
DATABASE_URL=postgresql+psycopg://agent_reviewer:agent_reviewer_local@127.0.0.1:5432/agent_reviewer \
REDIS_URL=redis://127.0.0.1:6379/0 \
uv run rq worker --url redis://127.0.0.1:6379/0 evaluations

Web workflow

Routes:

• GET / submission form
• POST /evaluations create or reuse an evaluation
• GET /evaluations/{id} result page
• POST /evaluations/{id}/retry retry a failed evaluation
• POST /evaluations/{id}/decision record a human decision
• DELETE /evaluations/{id}/personal-data purge retained personal data
• GET /api/evaluations/{id} strict JSON result or status payload
• GET /artifacts/{artifact_id} public result artifacts only
• GET /health/live
• GET /health/ready

Behavior notes:

• raw resume uploads are stored internally and are never downloadable from the public artifact route
• GitHub enrichment requires an explicit unchecked-by-default consent checkbox
• duplicate submissions reuse the existing evaluation instead of enqueuing another one
• failed evaluations can be retried as a new immutable attempt
• low-confidence or no_fit results block the human-decision action
• personal-data purge clears retained request PII and related private artifacts/checkpoints

Batch CSV evaluation imports

Routes:

• POST /evaluation-batches upload a candidate CSV and queue every valid repository as a batch
• GET /evaluation-batches/{batch_id} batch detail page with per-row status
• POST /evaluation-batches/{batch_id}/redrive retry undelivered batch rows

CSV format:

• required headers: Full Name, Email, Git Repository; an optional leading # index column and blank rows are ignored
• up to 500 data rows and BATCH_CSV_MAX_BYTES (default 1 MiB) per upload

Behavior notes:

• every row in a batch reuses the same operator-supplied job_description and evaluation_prompt; no LLM call runs during CSV import itself
• a well-formed file keeps its valid rows even when other rows are malformed or duplicated; invalid rows are recorded with a reason and never enqueued
• only newly created evaluations are enqueued — a repository URL already queued or completed under the same job/prompt/profile is reused, not re-run
• candidate Full Name and Email are operator-only batch metadata: they are never written into the EvaluationRequest, prompts, dossiers, artifacts, scores, or ranking inputs
• retained batch-row PII is purged on the same PII_RETENTION_DAYS schedule as other request PII
• batch routes require the same temporary HTTP Basic authentication as the rest of the web UI

Browser validation

Main browser flow:

npm run test:browser -- --project=chromium

Committed screenshot outputs:

• artifacts/ui/submission.png
• artifacts/ui/completed-evaluation.png
• artifacts/ui/ranking.png

README product-tour screenshots live in docs/images/ and are regenerated from the same fixture server:

• docs/images/submission.png
• docs/images/evaluation-result.png
• docs/images/batch.png
• docs/images/templates.png

Playwright MCP registration:

npx @playwright/mcp@0.0.78 --headless --output-dir artifacts/ui/mcp

MCP evidence captured in this worktree:

• artifacts/ui/mcp/completed-evaluation.png

Verification commands

Passing in this worktree during the final release pass:

uv run ruff format --check src tests scripts
uv run ruff check src tests scripts
uv run mypy src
uv run pytest tests/integration/test_web.py -q
uv run pytest tests/integration/test_pipeline_runtime.py -q
uv run pytest tests/integration/test_docker_runner.py tests/integration/test_resume_container.py tests/integration/test_github_enrichment.py tests/integration/test_queue_topology.py -q
uv run pytest tests/unit tests/integration -q --cov=agent_reviewer --cov-report=term-missing
uv run python -m agent_reviewer.web --check-startup
uv run python scripts/export_schema.py
uv run python -m agent_reviewer.profiles --validate profiles
npm run test:browser -- --project=chromium

Live-provider verification status:

• uv run pytest tests/integration/test_openhands_smoke.py -m live_openhands -q ran as 1 skipped, 2 deselected
• the authenticated OpenHands smoke is still blocked because this host has no configured OpenHands/OpenAI Codex subscription login; an API key is not required

Acceptance audit

Criterion	Evidence	Status
Minimal submission and result UI works	src/agent_reviewer/web.py, tests/integration/test_web.py, commit c3d5b14	Complete
Queued worker retries and checkpoint reuse are durable	src/agent_reviewer/worker.py, src/agent_reviewer/pipeline.py, tests/integration/test_pipeline_runtime.py, commit ae1cde3 plus 297addf	Complete
Missing provider auth fails closed	tests/integration/test_pipeline_runtime.py::test_worker_marks_terminal_failure_when_authentication_is_unavailable, commit 297addf	Complete
Browser flow and screenshots are reproducible	tests/browser/main-flow.spec.ts, playwright.config.ts, artifacts/ui/, commit a724643	Complete
Playwright MCP inspection recorded	artifacts/ui/mcp/completed-evaluation.png and completed-page MCP accessibility snapshot during final pass	Complete
Docker runner and PDF ingestion images build	final-pass docker build commands for Laravel runner, React Native runner, and PDF ingest image	Complete
Docker runner, resume container, GitHub enrichment, and queue topology integration tests pass	12 passed in final-pass integration run	Complete
Repo-wide Ruff lint is green	uv run ruff check src tests scripts	Complete
Repo-wide Ruff format check is green	uv run ruff format --check src tests scripts	Complete
Repo-wide mypy is green	uv run mypy src	Complete
Temporary internal HTTP Basic access control is enforced	tests/integration/test_web.py verifies missing, malformed, invalid, and valid credentials	Complete
Live authenticated OpenHands smoke completed	uv run pytest tests/integration/test_openhands_smoke.py -m live_openhands -q returned 1 skipped, 2 deselected because credentials were absent	Incomplete
Production queue burst topology verified against Redis/PostgreSQL worker execution	tests/integration/test_queue_topology.py plus final-pass run against disposable Dockerized Redis/PostgreSQL containers	Complete

Known remaining work before a broader production rollout

• run one real authenticated OpenHands smoke with persisted live evidence
• replace the temporary alpetg:alpetg Basic credential with a real identity provider before wider deployment