feat(plugin): Add Plugin evaluation support across all tiers

.github/workflows/ci.yml

@@ -281,6 +281,7 @@ jobs:
           tests/test_results_location_legacy_compatibility.py
           tests/test_tier3_compare.py
           tests/test_tier3_progress.py
+          tests/utils/test_secure_fs_platform.py
           tests/utils/test_secure_fs_selected_reads.py
           tests/test_oss_packaging.py
 

CHANGELOG.md

@@ -11,7 +11,9 @@ All notable changes to SkillEvaluator are documented in this file.
 - Initial public release candidate.
 - Enabled optional semantic-version validation in the default Tier 1 pipeline,
   including a public `--previous-version` monotonic-bump bound.
-
+- Added public plugin evaluation across all tiers: static schema and MCP checks,
+  advisory offline dependency/context deduplication, and Harbor-backed live
+  evaluation with effectiveness and optional sum-of-parts Integration arms.
 - Added NVIDIA Build live-agent paths: direct OpenCode support plus Docker
   compatibility bridges for Codex and experimental Claude Code, including
   multi-turn tool-call continuation.
@@ -21,10 +23,20 @@ All notable changes to SkillEvaluator are documented in this file.
 - Expanded the documentation site to fifteen pages — quickstart, eval
   datasets, agents and sandboxes, custom graders, reports, CI integration,
   CLI reference, and environment variables — under a task-oriented
-  navigation, with every command verified against the current CLI.
+  navigation, with every command verified against the current CLI. The public
+  guides now cover plugin evaluation flags, per-case input isolation, live
+  progress, partial-result safeguards, and context-aware quality scoring.
 
 ### Security
 
+- Hardened Tier 2 and plugin input handling with descriptor-anchored,
+  no-follow discovery and reads; linked, hard-linked, reparse-point, escaping,
+  and special files are rejected, while YAML/JSON nesting, scalar volume,
+  embedding work, and LLM prompt/response budgets are bounded before provider
+  calls or cache writes. The only redirect exception is the exact, contained
+  `CLAUDE.md -> AGENTS.md` public compatibility alias; Windows reads now use
+  native handle verification with the same fail-closed identity checks as
+  POSIX.
 - Secure Docker exec redaction now ignores environment values shorter than eight
   characters, matching the exact secret length floor used elsewhere. Short
   flags such as `CLAUDE_CODE_DISABLE_POLICY_SKILLS=1` no longer rewrite digits
@@ -90,6 +102,14 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ### Fixed
 
+- Kept Tier 3's interactive progress frame at a stable height, bounded visible
+  stage history, serialized terminal redraws, and safely disabled a reporter
+  when initialization or background refresh fails.
+- Made the `evaluate` workflow visible in top-level CLI help while retaining
+  the documented `skillevaluator tier3 evaluate` spelling.
+- Tier 3 preserves completed rewards from partially errored jobs only when each
+  aggregate error maps to a concrete failed trial; explicit failed statuses and
+  non-zero aggregate exit codes still suppress ambiguous scores.
 - Quality scoring now uses boundary-aware and context-aware matching for XML tags,
   reserved names, MCP guidance, README references, time references, exclusivity
   language, instruction action verbs, and nested Markdown links, avoiding
@@ -137,6 +157,9 @@ All notable changes to SkillEvaluator are documented in this file.
 - Programmatic dataset generation now returns explicit created, preview, and
   unchanged outcomes, preserves actionable failures, and no longer mutates
   process-wide command-line arguments.
+- Plugin manifest discovery is now root-bounded across all tiers, and Integration
+  evaluation requires explicit cross-component dataset evidence instead of
+  reporting unsupported composition claims.
 - Security and full-feature installs now work on RHEL 8 and other glibc 2.28
   Linux systems by keeping Semgrep and SkillSpector in separate tool
   environments while retaining compatible bundled Python dependencies.

THIRD_PARTY_NOTICES.md

@@ -1,12 +1,39 @@
 # Third-Party Notices
 
-This file lists the direct third-party Python dependencies declared by this
-distribution. `pyproject.toml` defines the dependency groups, and `uv.lock`
-records the exact resolved dependency set used for this release.
+This file lists direct third-party Python dependencies and vendored test
+fixtures in this distribution. `pyproject.toml` defines the dependency groups,
+and `uv.lock` records the exact resolved dependency set used for this release.
 
 | Dependency group | Packages and licenses |
 | --- | --- |
 | Base | Click (BSD-3-Clause), Jinja2 (BSD-3-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) |
 | LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) |
 | Tier 3 | Harbor (Apache-2.0) |
 | Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |
+
+## OpenClaw agent-skills test fixture
+
+`tests/fixtures/openclaw-autoreview/` contains pinned files from
+`openclaw/agent-skills`, commit `2a409d348a4bcf6f15e41e9a20efd0b298a32528`,
+path `skills/autoreview`. The source repository is licensed under the MIT
+License:
+
+Copyright (c) 2026 openclaw
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

docs/cli-reference.mdx

@@ -1,6 +1,6 @@
 ---
 title: "CLI Reference"
-description: "Every SkillEvaluator command, flag, and default — validate, quality-check, similarity-check, tier3 evaluate, and the tier1/tier2/tier3 expert groups."
+description: "Every SkillEvaluator command, flag, and default — validate, quality-check, similarity-check, tier3 evaluate, tier3 evaluate-plugin, and the tier1/tier2/tier3 expert groups."
 layout: reference
 ---
 
@@ -27,6 +27,7 @@ commands referenced below.
 | [dedup-scan](#dedup-scan) | Alias of `context-optimization-check` | Embeddings + LLM provider key |
 | [similarity-check](#similarity-check) | Detect duplicate content across a collection | Embeddings provider key |
 | [tier3 evaluate](#tier3-evaluate) | Run Tier 3 live agent evaluation | Provider key plus a credential the selected agents can use (one `NVIDIA_API_KEY` covers both on NVIDIA Build), backend (Docker by default) |
+| [tier3 evaluate-plugin](#tier3-evaluate-plugin) | Run public plugin effectiveness or Integration evaluation | The same Tier 3 runtime requirements as `tier3 evaluate` |
 | [create-eval-dataset](#create-eval-dataset) | Generate synthetic eval datasets | LLM provider key (none with `--no-llm`) |
 | [init-custom-grader](#init-custom-grader) | Scaffold a bring-your-own-grader starter | Nothing |
 | [init-harbor-task](#init-harbor-task) | Scaffold a bring-your-own-task Harbor starter | Nothing |
@@ -68,9 +69,9 @@ Most commands are also reachable under a tier-prefixed group — `skillevaluator
 | --- | --- |
 | `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` |
 | `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |
-| `tier3` | `evaluate`, `create-eval-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `validate`, `harbor-view` |
+| `tier3` | `evaluate`, `evaluate-plugin`, `create-eval-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `validate`, `harbor-view` |
 
-The exceptions run in both directions. `health-check` and [models](#models) are top-level only, with no tier-prefixed twin. And two commands are advertised **only** inside the `tier3` group: [tier3 evaluate](#tier3-evaluate), documented with its full flag set below, and `tier3 validate`:
+The exceptions run in both directions. `health-check` and [models](#models) are top-level only, with no tier-prefixed twin. [tier3 evaluate](#tier3-evaluate) is also discoverable as top-level `evaluate`, while [tier3 evaluate-plugin](#tier3-evaluate-plugin) and `tier3 validate` are advertised only inside the `tier3` group:
 
 ### tier3 validate
 
@@ -102,6 +103,8 @@ skillevaluator validate ./my-skill --tier3 -a codex \
     --block-on-agent-eval                                  # gate on Tier 3 too
 skillevaluator validate ./my-skill --autopilot            # Tier 3, generating evals if missing
 skillevaluator validate ./my-skill --full -a codex        # everything, one shot
+skillevaluator validate ./my-plugin --type plugin --tier3 \
+    --lift-mode both                                      # plugin effectiveness + Integration
 skillevaluator validate ./my-skill --tiers 1,3            # explicit tier selection
 skillevaluator validate ./skills-folder --full            # whole catalog, serially
 skillevaluator validate ./my-skill --tier3 -a codex,claude-code \
@@ -165,6 +168,7 @@ The following flags are forwarded to the live-eval engine **only when Tier 3 is
 | `--autopilot` | off | Generate an evaluation source automatically when missing, then run Tier 3 (implies `--tier3`). |
 | `-a, --agents TEXT` | `codex` | Comma-separated Harbor agents to evaluate. |
 | `--env-mode` | `docker` | Harbor environment backend (full list under [tier3 evaluate](#tier3-evaluate)). |
+| `--lift-mode [effectiveness\|integration\|both]` | `effectiveness` | Plugin only: compare the coordinated plugin with no plugin, its member skills staged individually, or both. Integration requires explicit cross-component dataset evidence. |
 | `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). |
 | `--n-concurrent INTEGER` | unset | Concurrent eval cases per agent. |
 | `--max-agents INTEGER` | unset | Maximum agents to run in parallel. |
@@ -298,9 +302,8 @@ Run Tier 3 live agent evaluation: real agents attempt your eval cases with and w
 skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
 ```
 
-<Note>
-The old top-level spelling `skillevaluator evaluate` still works for existing scripts, but no longer appears in `--help`; `tier3 evaluate` is the advertised name.
-</Note>
+The focused workflow is also discoverable as `skillevaluator evaluate`; both
+spellings share the same flags and behavior.
 
 | Flag | Default | Effect |
 | --- | --- | --- |
@@ -333,6 +336,54 @@ Without a `--model`/`--agent-model` override, each agent uses the selected provi
 
 Flags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults. The exception is `--results-dir`, which resolves independently of `evals/config.yml`.
 
+## tier3 evaluate-plugin
+
+Run Tier 3 against a public plugin without fetching remote components. The
+command accepts a bundle-reference `agent_plugin.yaml`/`.yml` target or a
+contained `.claude-plugin/plugin.json` target, stages the locally evaluable
+skills, rules, and MCP declarations into a temporary wrapper, and records any
+unresolved remote references in plugin provenance.
+
+```bash title="Evaluate plugin effectiveness and Integration"
+skillevaluator tier3 evaluate-plugin ./my-plugin --lift-mode both \
+  --evals-source ./evals/evals.json --agents codex
+```
+
+| Flag | Default | Effect |
+| --- | --- | --- |
+| `--evals-source PATH` | auto-detect | Dataset file, `evals/` directory, or skill/plugin directory containing `evals/`. |
+| `-a, --agents TEXT` | `codex` | Comma-separated Harbor agents. |
+| `--env-mode` | `docker` | Harbor environment backend; accepts the same values as [tier3 evaluate](#tier3-evaluate). |
+| `--skip-baseline` | off | Skip the no-plugin baseline. Invalid with `integration` or `both`, which require a baseline. |
+| `--lift-mode [effectiveness\|integration\|both]` | `effectiveness` | Compare the coordinated plugin with no plugin, its member skills staged individually, or both. `both` falls back to effectiveness when composition evidence is missing. |
+| `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). |
+| `--pass-threshold FLOAT` | unset | Score threshold (0.0–1.0) for a case to count as passed. |
+| `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. |
+| `--n-concurrent INTEGER` | unset | Concurrent eval cases per agent. |
+| `--max-agents INTEGER` | unset | Maximum agents to run in parallel. |
+| `--model TEXT` | unset | Global agent model override. |
+| `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable). |
+| `--custom-dockerfile-mode [preserve\|rebase]` | unset | Apply the dataset's custom Dockerfile using the same modes as `tier3 evaluate`. |
+| `--include-skills PATH` | none | Additional local member skills to stage (repeatable). Plugin evaluation always uses a group workspace. |
+| `--repo-root DIRECTORY` | auto-detect | Clone-root override used to resolve same-repository component references deterministically. |
+| `--copy-repo` | off | Copy the surrounding repository into the eval environment. |
+| `--grading-mode [default\|default_plus_custom\|custom_only]` | unset | Reward/grading mode — see [Custom Graders & Tasks](custom-graders.mdx). |
+| `--results-dir DIRECTORY` | unset | Directory for live-eval results. |
+| `--harbor-keep-jobs` | off | Retain Harbor job directories and artifacts for inspection. |
+| `--agent-runtime-preflight / --no-agent-runtime-preflight` | enabled | Run one bounded agent smoke task before the full evaluation matrix. |
+| `--timeout-multiplier FLOAT` | unset | Scale Harbor step timeouts. |
+| `--override-cpus INTEGER` | unset | Override eval-environment CPU allocation. |
+| `--override-memory-mb INTEGER` | unset | Override eval-environment memory in MB. |
+| `--override-storage-mb INTEGER` | unset | Override eval-environment storage in MB. |
+| `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run. |
+
+Integration requires at least one dataset case with `cross_component: true`
+and two or more distinct `expected_skills`. An explicit `integration` run is
+inconclusive without that evidence; `both` still runs the valid effectiveness
+comparison and records why Integration was skipped. See [Plugin
+evaluation](tier3-live-evaluation.mdx#plugin-evaluation) for component support,
+security restrictions, and result semantics.
+
 ## create-eval-dataset
 
 Create synthetic eval datasets for agent skill evaluation, written to `evals/evals.json`. Uses the configured LLM provider unless you pass `--no-llm`. Dataset anatomy and the four case buckets are documented in [Eval Datasets](eval-datasets.mdx).

docs/eval-datasets.mdx

@@ -47,7 +47,7 @@ Only the dataset is required. Everything else is optional and additive:
 | `evals/evals.json` | The eval dataset. Without it, live evaluation cannot run. | Yes — required |
 | `evals/config.yml` | Run policy: attempts, thresholds, runtime env, agent models, grading mode. | Yes |
 | `evals/EVAL.md` | Developer guidance that steers dataset generation. | Yes |
-| `evals/files/` | Input fixtures, staged into `/workspace/input/` in the container. | Yes |
+| `evals/files/` | Input fixtures. Legacy cases receive the full directory; cases with a `files` field receive only their declared fixtures. | Yes |
 | `evals/environment/` | Custom container environment: Dockerfile, sidecars, MCP servers. | Yes |
 | `evals/harbor/` | Bring-your-own-task native Harbor tasks — see [Custom Graders & Tasks](custom-graders.mdx). | Yes |
 | `evals/grader.py` or `evals/grader.sh` | Bring-your-own-grader script — see [Custom Graders & Tasks](custom-graders.mdx). | Yes |
@@ -86,13 +86,21 @@ top-level object with `skill_name` and an `evals` array.
 | `prompt` | Yes | The user request the agent receives. |
 | `expected_output` | Yes | Outcome-oriented description of what a successful agent produced. |
 | `assertions` | No | Observable agent behaviors to verify, one per entry. |
+| `files` | No | Relative fixture path or list of paths under `evals/`. When present, only these fixtures are staged into `/workspace/input/`; an empty or `null` value stages none. Omit the field to stage all of `evals/files/` for legacy datasets. |
 | `expected_skill` | No | Skill the agent should route to; `null` for negative cases. |
 | `expected_script` | No | Script the agent should run; `null` if none applies. |
 | `acceptable_skills` | No | Closely related skills that earn partial routing credit (legacy alias: `acceptable_alternates`). |
 
 The dataset file can be named `evals.json`, `evals.jsonl`, `evals.yaml`, or
 `evals.yml` — `tier3 evaluate` accepts all four.
 
+<Note>
+Declare `files` on new cases so each trial receives only the fixtures it needs.
+This prevents negative or unrelated cases from seeing other cases' inputs.
+Paths are resolved within `evals/`; absolute paths, URI schemes, and traversal
+outside the directory are rejected before the trial is staged.
+</Note>
+
 <AccordionGroup>
   <Accordion title="Legacy flat format">
     Older datasets are a flat array of entries with `question` instead of