Skip to content

evaluation.md

Latest commit

 

History

History
138 lines (110 loc) · 6.23 KB

evaluation.md

File metadata and controls

138 lines (110 loc) · 6.23 KB
Raw

PaperRefine Evaluation Record

Purpose

This record separates reproducible repository checks from model-behavior observations. Static tests protect the Skill contract, trigger boundaries, packaging, and preservation fixtures. Model evaluations remain probabilistic and must never be reported as deterministic tests.

Scenarios

The redesign uses five representative scenarios:

  1. An English Results paragraph containing 18.4%, p = 0.041, a named citation, and the qualifier may.
  2. A Chinese Discussion paragraph that claims causality and universal generalization from a retrospective observational study.
  3. An already strong abstract that should receive no unnecessary novelty or synonym edits.
  4. A LaTeX sentence containing \ref{fig:main}, 95\%, $p < 0.05$, and \cite{smith2025}.
  5. A compound near-miss request for full-paper drafting, literature search, reviewer response, and paper-to-PPT generation.

Evaluation dimensions are trigger accuracy, protected-token preservation, claim calibration, naturalness, output-contract compliance, and unnecessary interaction.

Dynamic Baseline Attempt

Date: 2026-07-16

Five isolated no-Skill Codex CLI sessions were started with ephemeral, read-only workspaces and a temporary CODEX_HOME. The CLI repeatedly lost the model stream, exhausted five WebSocket retries, fell back to HTTP, and timed out. All five processes were interrupted after the fallback also failed. No final model responses were produced, so there is no valid dynamic baseline to score.

The LaTeX prompt also demonstrated that inline shell quoting is not a reliable evaluation transport for $p; future dynamic evaluation must provide the prompt through a literal file or stdin. This transport issue is not attributed to either the model or the Skill.

Legacy Skill Static Baseline

The committed nature-polishing Skill exhibits these measurable contract problems before redesign:

  • Frontmatter contains version and author in addition to the portable name and description fields.
  • The Skill body is 386 lines and mixes routing, a writing course, section advice, mechanics, ethics, and output behavior in one file.
  • The identity and default style are coupled to Nature even though most requested academic revision is venue-neutral.
  • It uses rigid sentence-length targets and British spelling defaults that are not universal across fields or venues.
  • It provides five overlapping references but no executable trigger, packaging, link, or preservation checks.
  • It lacks explicit negative trigger boundaries for drafting, search, reviewer response, and paper-to-PPT requests.
  • Its side-by-side output is optional and does not classify changes by author verification risk.

Forward Evaluation

Date: 2026-07-16

Installation smoke tests

Four isolated installation checks passed:

  1. A direct Codex Skill copy containing only SKILL.md and references/ resolved all six conditional references relative to the installed Skill.
  2. The Claude Code marketplace entry resolved its plugin manifest and packaged skills/paper-refine/SKILL.md entry.
  3. A copied plugin archive remained self-contained, with both platform manifests and all six references available without the repository root.
  4. Codex CLI 0.144.0 successfully added the repository as a local marketplace in a temporary CODEX_HOME, installed paper-refine@paper-refine, and reported version 0.1.0 as installed and enabled. No user plugin or Skill configuration was changed.

These checks establish packaging and relative-path integrity. They do not score the quality of a model rewrite.

Dynamic outcome

The isolated Codex CLI model channel remained unavailable: it exhausted five WebSocket retries, fell back to HTTP, and timed out for both the no-Skill and installed-Skill attempts. These failed attempts produced no output and are not scored.

Claude Code successfully ran four cases with a temporary HOME and workspace. Only sanitized user-level authentication/model settings were copied; project/local settings, other user Skills/plugins, memory, and MCP servers were excluded. The built-in Skill loader was the only enabled tool. The generated per-case metadata records the exact CLI version and model:

  1. Explicit claim calibration narrowed observational causality and universal generalization without adding confounding, cohort structure, or future study designs.
  2. Explicit minimal-change editing returned an already strong abstract byte for byte in Refined text.
  3. Explicit LaTeX editing preserved \ref{fig:main}, 95\%, $p < 0.05$, and \cite{smith2025}.
  4. A natural-language near miss requesting full-paper drafting, literature search, reviewer response, and PPT did not return the PaperRefine contract.

Early claim-calibration runs added common but unstated methodological limitations. Those failures drove a stricter evidence boundary and a whole-output grader: unstated limitations, evidence forms, analyses, and study designs are rejected in Refined text, explanations, and author checks alike. Claim-calibration semantics are also checked inside Refined text, so safe keywords elsewhere cannot mask an unsafe rewrite. The committed files are the final isolated run after those criteria and the Skill contract were fixed.

Raw outputs and generated run metadata are stored under evals/results/2026-07-16/. scripts/grade_forward_eval.py verifies the behavioral criteria plus prompt, output, and loaded Skill-bundle hashes. Run metadata records the names, but never the values, of sanitized user-setting and process environment variables. The grader reports 4/4 passing and is run in CI. Mutation tests verify that it fails for missing protected tokens, invented content anywhere in the response, unsafe claim semantics, unexpected PaperRefine routing, unsafe or colliding paths, leaked environment keys, and tampered metadata.

To regenerate model outputs deliberately:

python scripts/run_claude_forward_eval.py --confirm-cost
python scripts/grade_forward_eval.py

The runner never calls a model without --confirm-cost. The evidence is one run per case, not a statistical reliability estimate; future releases should add repeat runs and more domain-diverse cases rather than treating 4/4 as a guarantee for all manuscripts.