Define task-specific local agent and model evaluation profiles

[szmyty]

Outcome

Define versioned evaluation profiles that determine whether a local agent/model configuration is good enough for a specific workflow role, rather than treating installation, model size, or generic benchmark scores as proof of quality.

This complements #64 capability compatibility, #65 workflow contracts, Hygiene's claims policy, and Observatory's comparison surfaces.

Evaluation model

Represent:

• task/workflow/node identity and version;
• required versus optional capabilities;
• model, provider/runtime, quantization, prompt/skill, tool, retrieval, and generation configuration;
• hardware, OS, architecture, container/runtime, and resource limits;
• fixture/corpus identity, license, sensitivity, and provenance;
• deterministic and human-assessed metrics;
• quality thresholds, safety guardrails, no-harm gates, and known-negative cases;
• latency, memory, disk, startup, throughput, energy proxy, and context limits;
• supported, partial, blocked, failed, superseded, and unknown decisions;
• raw evidence, represented revision, freshness, and reproducibility.

Initial role profiles

Use synthetic/public fixtures to define bounded profiles for:

• routing/classification;
• document extraction and structured normalization;
• retrieval and query expansion;
• summarization and evidence synthesis;
• citation/provenance verification;
• code planning/review;
• confidential legal-document issue spotting;
• optional multimodal/media workflow nodes.

A single model may qualify for some roles and fail others.

Selection rules

• Capability eligibility is checked before quality.
• Routing consumes accepted evidence; it does not invent support.
• Higher quality does not override privacy, license, hardware, cost, or policy constraints.
• Confidential roles exclude hosted execution unless a distinct policy explicitly permits it.
• Failed and negative results remain preserved.
• Thresholds are declared before candidate results are examined.
• Human assessments identify rubric, assessor count, disagreement, and adjudication.

Acceptance criteria

• A versioned task-evaluation profile and result schema exist.
• Capability, quality, safety, resource, privacy, license, and support state remain distinct.
• At least three materially different local models/configurations are evaluated on two roles, or unavailable candidates are documented.
• One model qualifies for one role and fails/blocks/unknowns another to prove non-global ranking.
• Synthetic confidential fixtures never contain real personal data.
• Deterministic metrics and human judgments retain complete provenance.
• Routing/selection output is generated from accepted evidence.
• Reproduction, freshness, supersession, and stale-result behavior are defined.
• Observatory can visualize results without receiving private prompts or payloads.
• Documentation prevents unsupported “best model” or equivalence claims.

Dependencies / related

• Define an evidence-backed quantitative and compatibility claims policy hygiene#30
• Define an evidence-backed agent capability manifest #64
• Define governed operational-agent workflows inspired by Teambridge #65
• Publish provenance-preserving benchmark and evaluation comparisons observatory#14
• Provide an OpenAI-compatible local model gateway profile realm#27
• Publish an evidence-grounded legal document review skill #72
• Roadmap: build the local-first agent workstation and confidential research environment realm#25

Non-goals

• Creating one universal model leaderboard.
• Treating cloud and local configurations as equivalent without comparable evidence.
• Downloading every candidate model.
• Using generated text as ground truth for legal, medical, financial, or safety claims.