Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,22 @@ help:
@echo " make ci lint + test (CI gate)"
@echo " make clean Remove outputs/ (except CSV pilot panel)"

agent-check: claim-check doc-links-check
@echo "All agent checks passed."

claim-check:
@echo "--- Claim language scan (advisory) ---"
PYTHONPATH=src $(PYTHON) scripts/safety/check_claims.py
@echo "OK"

claim-check-strict:
@echo "--- Claim language scan (strict, will fail on findings) ---"
PYTHONPATH=src $(PYTHON) scripts/safety/check_claims.py --fail-on-findings

doc-links-check:
@echo "--- Doc link check ---"
PYTHONPATH=src $(PYTHON) scripts/check_doc_links.py
@echo "OK"

demo:
PYTHONPATH=src $(PYTHON) -m openamp_foundry.cli rank \
Expand Down
8 changes: 4 additions & 4 deletions docs/engineering/CI_AND_QUALITY_GATES.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,19 +132,19 @@ Agent-generated changes should verify:

## Local command targets to add over time

Future useful commands:
Available commands:

```text
make docs-check # internal links, source-of-truth references, index coverage
make docs-check # internal links, source-of-truth references, index coverage (alias: doc-links-check)
make claim-check # forbidden or risky claim language
make agent-check # safe-scope and agent PR checks (runs claim-check + doc-links-check)
make artifact-check # schema/version/release-status validation
make release-check # release checklist automation
make agent-check # safe-scope and agent PR checks
make data-check # dataset-card and license metadata validation
make model-check # model-card and release-status validation
```

These should start as advisory checks before becoming hard gates.
`agent-check`, `claim-check`, and `doc-links-check` are implemented. Others should start as advisory checks before becoming hard gates.

## Failure reports

Expand Down
19 changes: 19 additions & 0 deletions docs/evidence/GUIDE_CLI_PARSER_ORGANIZATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# CLI Parser Construction into Command Modules

Guidelines for organizing CLI parser construction.

## Current Structure
```
cli/main.py — Main parser and command routing
cli/commands/ — Command handler modules
benchmark.py — Bench-related handlers
selection.py — Selection-related handlers
reports.py — Report-related handlers
```

## Guidelines
- Each command group should have its own module in `cli/commands/`.
- The main parser in `main.py` should only register subparsers and route to handlers.
- Handler modules should not modify the parser after registration.
- New command groups should follow the same pattern.
- Keep handler functions focused — one function per command.
27 changes: 27 additions & 0 deletions docs/evidence/GUIDE_FAILURE_REPORT_FORMATTER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Failure Report Formatter for Local Checks

Formats failure reports from local validation checks.

## Output Format
```json
{
"check": "check_doc_links",
"status": "failed",
"failed": 5,
"passed": 95,
"total": 100,
"errors": [
{"file": "docs/example.md", "error": "Broken link: PROOF_LADDER.md"}
],
"suggested_actions": [
"Fix the broken links in docs/example.md",
"Run the check again to verify"
]
}
```

## Rules
- Failure reports should be machine-readable (JSON).
- Include the check name, status, and error details.
- Include suggested actions for fixing common errors.
- Reports should be written to `outputs/check_reports/`.
28 changes: 28 additions & 0 deletions docs/evidence/GUIDE_GATE_PROMOTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Advisory-to-Blocking Gate Promotion Records

Records of gates being promoted from advisory to blocking.

## Record Format
```json
{
"gate": "benchmark_regression",
"promoted_at": "2026-07-08",
"reason": "Benchmark regression > 0.02 now blocks CI",
"previous_status": "advisory",
"new_status": "blocking",
"decision_log_ref": "DR-2026-07-08-001"
}
```

## Current Gate Statuses
| Gate | Status | Description |
|------|:------:|-------------|
| Benchmark regression | blocking | AUROC drop > 0.02 fails CI |
| Doc link check | advisory | Broken links are reported, don't block |
| Claim check | advisory | Overclaiming language is reported |
| Test suite | blocking | Test failures block CI |

## Rules
- Promoting a gate from advisory to blocking requires a decision log entry.
- Demoting a gate from blocking to advisory requires project lead approval.
- Gate status changes should be announced in release notes.
22 changes: 22 additions & 0 deletions docs/evidence/GUIDE_PATH_RESOLVER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Repository Path Resolver Utility

Helper for resolving paths within the repository.

## Usage
```python
from pathlib import Path

def repo_path(*parts: str) -> Path:
"""Resolve a path relative to the repository root."""
return Path(__file__).resolve().parent.parent / Path(*parts)

# Examples
candidates = repo_path("examples", "sequences", "demo_candidates.csv")
config = repo_path("configs", "pipeline.yaml")
```

## Rules
- Always resolve paths relative to the repository root, not the current working directory.
- Use `Path.resolve()` to handle symlinks and relative paths.
- Never hardcode absolute paths.
- Test paths should use `tmp_path` fixture where possible.
19 changes: 19 additions & 0 deletions docs/evidence/REGISTRY_COMMAND_ARTIFACTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Command Artifact Type Registry

Type of artifact produced by each command.

| Command | Artifact Type | Format | Location |
|---------|:-------------|:------|----------|
| rank | ranked_candidates | JSONL | outputs/*_ranked.jsonl |
| rank | evidence_certificate | JSON | outputs/evidence/ |
| rank | run_manifest | JSON | outputs/run_manifest.json |
| validate | validation_result | stdout | — |
| bench | benchmark_result | JSON | outputs/*.json |
| pilot-panel | panel_csv | CSV | outputs/*_panel.csv |
| calibration-intake | intake_report | JSON | outputs/*_intake.json |
| recalibration-gate | gate_verdict | JSON | outputs/*_verdict.json |

## Rules
- Every command should produce at least one artifact.
- Artifact types should be registered here.
- If a command's output format changes, update this registry.
18 changes: 18 additions & 0 deletions docs/evidence/REGISTRY_COMMAND_HANDLERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Command Handler Registry

Registry of CLI command handlers and their locations.

| Command | Handler Module | Handler Function |
|---------|---------------|-----------------|
| rank | cli/main.py | `run_ranking_pipeline` |
| validate | cli/main.py | `validate_json_schema` |
| bench | cli/commands/benchmark.py | Various `_run_*` functions |
| pilot-panel | cli/commands/selection.py | `_run_pilot_panel` |
| calibration-intake | cli/commands/reports.py | Various `_run_*` functions |
| recalibration-gate | cli/commands/reports.py | Various `_run_*` functions |

## Rules
- New commands should be registered here.
- Each command should map to exactly one handler function.
- Handlers should be in `cli/commands/` or `cli/main.py`.
- If a handler is moved, update this registry.
20 changes: 20 additions & 0 deletions docs/evidence/REGISTRY_COMMAND_SAFETY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Command Safety Class Registry

Safety classification for CLI commands.

| Command | Safety Class | Rationale |
|---------|:-----------:|-----------|
| rank | safe | Scores candidates, no external effects |
| validate | safe | Validates files, read-only |
| bench | safe | Runs benchmarks, read-only |
| calibration-intake | requires_review | Affects calibration state |
| recalibration-gate | requires_review | Affects calibration decisions |
| pilot-panel | safe | Selection algorithm, read-only |
| lab-result-report | safe | Report generation, read-only |

## Classes
| Class | Meaning | Required Checks |
|-------|---------|----------------|
| safe | No harmful effects possible | Standard CI |
| requires_review | Could affect decisions | Human review required |
| restricted | High-risk operation | Project lead approval |
15 changes: 15 additions & 0 deletions tests/test_cli_entrypoint.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
"""Test CLI entrypoint."""


def test_main_help():
import subprocess
import sys
r = subprocess.run([sys.executable, "-m", "openamp_foundry.cli", "--help"],
capture_output=True, text=True, env={"PYTHONPATH": "src"})
assert r.returncode == 0
assert "usage" in r.stdout.lower() or "usage" in r.stderr.lower()


def test_main_version():
from openamp_foundry import __version__
assert len(__version__) > 0
27 changes: 27 additions & 0 deletions tests/test_csv_input_contract.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
"""Test CSV input contract validation."""
from pathlib import Path
import csv


def test_required_columns_present():
"""CSV files should have required columns."""
with open("examples/sequences/demo_candidates.csv", newline="") as f:
reader = csv.DictReader(f)
headers = reader.fieldnames
required = {"id", "sequence"}
assert required.issubset(set(headers)), f"Missing columns: {required - set(headers)}"


def test_sequences_are_strings():
with open("examples/sequences/demo_candidates.csv", newline="") as f:
for row in csv.DictReader(f):
assert isinstance(row["sequence"], str)
assert len(row["sequence"]) > 0


def test_ids_are_unique():
ids = []
with open("examples/sequences/demo_candidates.csv", newline="") as f:
for row in csv.DictReader(f):
ids.append(row["id"])
assert len(ids) == len(set(ids))
17 changes: 17 additions & 0 deletions tests/test_package_import_smoke.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
"""Test package import for installed mode."""
import subprocess
import sys


def test_core_import():
r = subprocess.run([sys.executable, "-c", "import openamp_foundry; print(openamp_foundry.__version__)"],
capture_output=True, text=True, env={"PYTHONPATH": "src"})
assert r.returncode == 0
assert len(r.stdout.strip()) > 0


def test_cli_import():
r = subprocess.run([sys.executable, "-c", "from openamp_foundry.cli import main; print('ok')"],
capture_output=True, text=True, env={"PYTHONPATH": "src"})
assert r.returncode == 0
assert r.stdout.strip() == "ok"
Loading