eval: matrix.py build_cfg_template(warm=True) copies skills from a nonexistent path and silently produces a skill-less cfg

[toejough]

Observed

dev/eval/cumulative/matrix.py::build_cfg_template(warm=True) copies skills from a nonexistent path and silently no-ops instead of raising an error.

Repro

1. Inspect matrix.py lines 84-89:

if warm:
    # real.full: both /recall and /learn skills from the repo (the shipped skills).
    for skill in ("recall", "learn"):
        src = os.path.join(REPO, "skills", skill)
        if os.path.isdir(src):
            shutil.copytree(src, os.path.join(dst, "skills", skill))

2. Run ls /Users/joe/repos/personal/engram/skills 2>&1:

ls: /Users/joe/repos/personal/engram/skills: No such file or directory

3. Run ls /Users/joe/repos/personal/engram/agent-instructions/skills:

agent-instructions/skills/
├── curate
├── learn
├── please
├── recall
├── route
└── write-memory

4. Repro via Python one-liner: create a temporary directory and call build_cfg_template:

import tempfile
import sys
import os
sys.path.insert(0, '/Users/joe/repos/personal/engram/dev/eval/cumulative')
import matrix
with tempfile.TemporaryDirectory() as tmpdir:
    matrix.build_cfg_template(tmpdir, warm=True)
    print("skills dir exists?", os.path.isdir(os.path.join(tmpdir, "skills")))
    print("contents:", os.listdir(tmpdir))
# Output: skills dir exists? False, contents: ['.claude.json']

The source path is hardcoded to REPO/skills/<skill>, which does not exist. The guard if os.path.isdir(src) silently skips the copy, producing a cfg with no recall/learn skill.

Expected

1. The source path should be agent-instructions/skills/<skill> (the actual location where skills live since they moved).
2. A missing source directory should raise a hard error, not silently skip. This prevents harnesses from running with a crippled "warm" configuration that has no recall skill.

Context

Any harness that relied on warm=True since the skills moved (likely since ~2026-08-23 based on agent-instructions structure changes) may have run "warm" arms without the recall skill. Prior warm results should be checked for "skill":"recall" in their transcripts to identify affected runs.

The workaround/corrected version exists in dev/eval/cumulative/runbook_vs_skill/probe.py::build_cfg_template (lines 143-170), which correctly uses agent-instructions/skills as the source and is intentionally a LOCAL reimplementation since matrix.py is out of scope to edit from that eval task.

Affected files

• dev/eval/cumulative/matrix.py (lines 84-89: wrong source path + silent skip on missing source)

Related

• eval: real-vault fingerprint guard alarms on the orchestrator's own vault writes (false ABORT-REPORT) #750
• dev/eval: probe harnesses don't isolate the vault — trial agents write real notes into the operator's vault #708