Skip to content

Define composable audit profiles and promote the repository-audit skill #86

Description

@szmyty

Outcome

Evolve Aether's existing approved auditor specification and draft audit-repository skill into a canonical, composable audit-profile system that supports focused audits and a master holistic audit without duplicating agent logic.

This issue does not create another auditor agent. Aether already owns the reusable auditor specification, skill, and agent. The goal is to separate audit intent/profile from the generic execution procedure so a user or downstream orchestration system can say "run a feature-opportunity audit" or "run the holistic profile" and receive a predictable evidence-backed report.

Existing foundation to preserve

  • library/organization/specs/quality/auditor.spec.md is the approved repository-audit contract.
  • library/organization/skills/quality/audit-repository/SKILL.md is the canonical draft execution skill.
  • library/organization/agents/auditor/AGENT.md is the reusable read-only agent.
  • Historical audit reports remain immutable source evidence under consumer-owned audits/ directories.

Do not fork these into repository-specific copies.

Profile model

Define a versioned audit-profile contract with at least these initial profiles:

feature-opportunity

Search for evidence-backed opportunities for useful new capabilities, integrations, workflows, ergonomics, or product improvements. Distinguish a plausible opportunity from an actual requirement.

experimental-opportunity

Identify deliberately speculative or research-oriented ideas that may be worth prototyping. Experimental findings must be clearly separated from production gaps and cannot become blocking conformance failures by default.

implementation-quality

Compose code quality, bugs, reliability, testing, CI/CD, security/privacy, performance, maintainability, documentation, and developer-experience checks as applicable.

contract-drift

Look for missing required artifacts, stale/generated drift, spec-to-implementation mismatch, ownership violations, obsolete assumptions, missing integrations, and capabilities that previously existed or were intended but have drifted away.

holistic

Compose all applicable stable profiles into one comprehensive repository assessment while preserving the source profile for every finding.

Profiles should remain extensible so future domains such as release readiness, accessibility, publication quality, or dependency health can be registered without changing the core finding schema.

Design requirements

  • Keep the specification authoritative for audit semantics and finding/report contracts.
  • Keep the skill focused on the reusable execution procedure.
  • Keep the agent thin: resolve the selected profile(s), consume the spec/skill, gather evidence, and write reports.
  • Profiles define focus, applicability, required evidence classes, exclusions, and composition; they must not become giant duplicated prompts.
  • Support one profile, several explicitly selected profiles, or a named composite.
  • Record the exact profile IDs/versions used in every report.
  • Preserve observed / inferred / recommended / unverified distinctions.
  • Preserve read-only behavior by default.
  • Candidate GitHub issues may be suggested in reports, but issue creation remains a separate explicitly authorized workflow.
  • Unknown, partial, blocked, unsupported, and not-applicable states must remain first-class.
  • Experimental recommendations must never be rendered as confirmed defects merely because they are absent.

Machine-readable metadata

Extend the audit output contract so downstream systems can normalize at least:

  • audit ID and profile IDs/versions;
  • represented repository/revision;
  • started/completed timestamps;
  • scope and exclusions;
  • completion state;
  • finding IDs;
  • classification, severity, confidence, status, effort, impact;
  • evidence references;
  • source profile;
  • candidate issue metadata when present;
  • predecessor/prior-audit relationships where known.

Prefer a deterministic machine-readable sidecar or projection from the canonical report rather than requiring dashboards to scrape prose.

Skill maturity

Promote audit-repository from draft only after:

  • the staged/canonical delta is reconciled;
  • the stale internal issue reference in the skill is corrected;
  • profile selection/composition is covered by deterministic fixtures;
  • the skill/spec/agent dependency graph validates.

Acceptance criteria

  • A versioned audit-profile schema/registry exists in canonical Aether source.
  • feature-opportunity, experimental-opportunity, implementation-quality, contract-drift, and holistic profiles are defined.
  • holistic composes other profiles rather than re-copying their instructions.
  • Every finding preserves its originating profile and evidence classification.
  • Stable gaps, speculative opportunities, and policy/conformance failures cannot be conflated.
  • Reports expose deterministic machine-readable metadata suitable for Observatory ingestion.
  • Historical audit reports remain immutable.
  • audit-repository is reconciled with staged material and its stale internal references are fixed.
  • The auditor agent remains read-only and thin over canonical spec/skill/profile inputs.
  • Deterministic fixtures cover single-profile, multi-profile, holistic, partial, blocked, and not-applicable runs.
  • Aether validation/catalog/projection/evaluation checks pass.

Related work

  • Historical universal auditor: egohygiene/egohygiene#315
  • First-party reusable agents: egohygiene/aether#11
  • Repository Intelligence: egohygiene/relay#27
  • Organization/fleet read models: egohygiene/observatory#7

Non-goals

  • Running audits across the whole organization in this issue.
  • Creating GitHub issues automatically from findings.
  • Implementing a dashboard in Aether.
  • Moving conformance policy from Hygiene into Aether.
  • Giving the auditor mutation authority.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions