RFC: Artifact Health & Quality Gates (SonarQube-inspired)

[brandonrc]

Summary

Build a unified artifact health scoring and quality gate system inspired by SonarQube, integrating security scanning, license compliance, format-specific linting, and best-practice checks into a single health score per artifact and repository.

Motivation

We already have strong foundations:

• Trivy + Grype + OpenSCAP scanning (5+ scanners)
• Dependency-Track integration for SBOM analysis
• Promotion policies with CVE thresholds + license gates
• Repository security scores (A-F grading)

But these are siloed. Users have to check security scores in one place, license compliance in another, and there are no format-specific quality checks (Helm chart validation, Dockerfile best practices, protobuf breaking changes, etc.). There is no single "is this artifact healthy?" answer.

Proposed Architecture

Quality Dimensions

Dimension	Source	Example Checks
Security	Trivy, Grype, DTrack	CVE counts by severity, known exploits
License Compliance	SBOM analysis	Denied licenses, unknown licenses
Format Lint	Format-specific tools	helm lint, hadolint, buf lint, tflint
Best Practices	Built-in rules	Metadata completeness, version hygiene, signed artifacts
Breaking Changes	Diff analysis	API compatibility (protobuf, OpenAPI), semver compliance

Unified Health Score

Like SonarQube's quality gate:

• A-F grade per artifact (weighted composite of all dimensions)
• Repository-level score (aggregate of all artifacts)
• Quality gate pass/fail for promotion decisions
• Trend tracking (score over time, regression detection)

Quality Check Pipeline

Artifact Upload → AfterUpload hook → Enqueue quality checks
                                           ↓
                              Background worker (10s tick)
                                           ↓
                              Run format-specific checkers
                                           ↓
                              Persist results + update scores
                                           ↓
                              Quality gate evaluation

Format-Specific Checks (Phase 1)

Format	Tool	Checks
Helm	helm lint, kubeconform	Chart validation, K8s schema, deprecated APIs
Docker/OCI	hadolint	Dockerfile best practices, base image age
Protobuf	buf lint, buf breaking	Style violations, API breaking changes
Terraform	tflint, tfsec	HCL validation, security rules
Python (PyPI)	pip-audit, metadata check	Known vulns, metadata completeness
npm	npm audit	Dependency vulnerabilities

Database Schema (New Tables)

-- Per-artifact quality results
artifact_quality_checks (
  id, artifact_id, check_type, checker_name,
  status, score, issues_count,
  critical_count, warning_count, info_count,
  report JSONB, started_at, completed_at
)

-- Materialized artifact health score
artifact_health_scores (
  artifact_id UNIQUE, overall_score, grade,
  security_score, license_score, lint_score,
  best_practice_score, quality_gate_passed,
  last_checked_at
)

-- Repository-level aggregate
repo_health_scores (
  repository_id UNIQUE, overall_score, grade,
  artifact_count, healthy_count, failing_count,
  trend_direction, calculated_at
)

-- Quality gate definitions
quality_gates (
  id, repository_id, name,
  min_score, required_checks,
  block_on_fail, conditions JSONB
)

API Endpoints

GET  /api/v1/artifacts/{id}/health          → ArtifactHealthResponse
GET  /api/v1/repositories/{key}/health      → RepoHealthResponse
GET  /api/v1/health/dashboard               → Global health dashboard
POST /api/v1/artifacts/{id}/quality-check   → Trigger on-demand check
GET  /api/v1/quality-gates                  → List quality gate definitions
POST /api/v1/quality-gates                  → Create quality gate

Integration with Existing Systems

• Promotion policies gain a new dimension: quality gate must pass before promotion
• Security scanning results feed into the security dimension of health scores
• SBOM/license data feeds into the compliance dimension
• Sync policies could filter by health score (only replicate healthy artifacts)

Questions for Discussion

1. Should quality checks block uploads (synchronous) or run async and gate at promotion time?
2. What format-specific checks would be most valuable for your workflows?
3. Should health scores be public (visible without auth) or follow repo permissions?
4. Interest in custom quality gate rules (e.g., "no artifacts older than 90 days in production repo")?
5. Should we support pluggable checkers via WASM plugins, or keep them as built-in Rust modules?

[brandonrc]

Implementation Complete

This RFC has been fully implemented across backend and web frontend. Here's a summary of what shipped:

Backend (artifact-keeper)

Quality Check Service (quality_check_service.rs)

• 4-dimension health score: Security (40%), Quality (25%), License (20%), Metadata (15%)
• Adaptive weighting — missing dimensions are excluded and weights redistribute
• A-F letter grading (same scale as existing security scores)
• Built-in checkers: metadata completeness (all formats), Helm lint (Helm charts)
• Async check execution with background worker
• Issue suppression with audit trail

Quality Gates API (quality_gates.rs)

• Full CRUD for quality gate definitions (global + per-repository)
• Gate evaluation with min score thresholds, max issue counts, required checks
• Three enforcement actions: allow, warn, block
• Integrated into the promotion pipeline — gates evaluate before CVE/license policies

API Endpoints (all under /api/v1/quality/):

• GET /health/artifacts/{id} — per-artifact health with component breakdown
• GET /health/repositories/{key} — repo-level aggregate health
• GET /health/dashboard — org-wide dashboard (admin)
• POST /checks/trigger — trigger checks on artifact or repo
• GET /checks?artifact_id={id} — list check results
• GET /checks/{id}/issues — list issues from a check
• POST /issues/{id}/suppress / DELETE /issues/{id}/suppress
• GET/POST/PUT/DELETE /gates — quality gate CRUD
• POST /gates/evaluate/{artifact_id} — on-demand gate evaluation

Promotion Integration (promotion.rs, promotion_policy_service.rs)

• Quality gates are the first layer in the 3-stage promotion pipeline: Quality Gate → CVE Policy → License Policy
• Blocking gates reject promotion before policy checks run
• skip_policy_check: true bypasses all gates (logged for audit)
• Rejection workflow with status tracking and rejection reasons

Web Frontend (artifact-keeper-web)

Health Dashboard (admin page at /admin/health)

• Organization-wide health score with circular progress + letter grade
• Grade distribution visualization (horizontal segment bar)
• Key metric cards (total repos, artifacts evaluated, grade A count, failing count)
• Sortable repository health table with all component scores

Artifact Health Tab (artifact detail page)

• Overall health score + grade
• Component score breakdown with progress bars and weight labels
• Check results list with pass/fail status
• Issues list with severity badges and suppression controls

Quality Gates Admin (admin page)

• Create, edit, delete gates
• Rule configuration UI with score thresholds and issue limits
• Toggle between allow/warn/block enforcement

Promotion Dialog (staging view)

• Gate evaluation results shown inline
• Pass/fail indicators per artifact
• Violation details with skip option (admin only)

Documentation

New docs pages added to the Starlight site (artifact-keeper-site PR #7):

• Artifact Health Scores — dimensions, weights, API reference, web UI overview
• Quality Gates — gate rules, enforcement, promotion integration, examples
• Updated Staging & Promotion — documents the 3-layer gate pipeline

What's Deferred

Per the original RFC's "Phase 1" scope, additional format-specific checkers (hadolint for Docker, buf lint for protobuf, tflint for Terraform) are not yet implemented. The architecture supports adding them — each checker is a module that returns a score + issues list. These can be tracked as follow-up issues.

Trend tracking (score over time, regression detection) is also deferred.

Addressing the Discussion Questions

1. Sync vs async? → Async. Checks run in the background and gate at promotion time.
2. Format-specific checks? → Metadata completeness (all formats) and Helm lint shipped. More checkers can be added incrementally.
3. Public or permissioned? → Health scores follow repo permissions.
4. Custom gate rules? → Yes — gates support min scores, max issue counts, and required checks per repository.
5. WASM plugin checkers? → Not yet, but the checker interface is modular enough to support it later.

Closing this as implemented.