Add document adapters and evaluate the SourcePack workflow

README.md

@@ -2,7 +2,7 @@
 
 **Give your coding agent a source. Get an explanation you can check and return to.**
 
-SourcePack is an installable skill for inspecting YouTube videos, public webpages and local text. It coordinates existing extraction tools, asks the host agent to read the text and inspect video frames, and saves observations against their original evidence.
+SourcePack is an installable skill for inspecting YouTube videos, public webpages, local text and optional PDF/Office documents. It coordinates existing extraction tools, asks the host agent to read the text and inspect video frames, and saves observations against their original evidence.
 
 ```text
 “Use SourcePack to explain this tutorial. Check the settings shown on screen,
@@ -23,11 +23,13 @@ python3 scripts/install.py --skills-dir ~/.codex/skills
 
 For other skill-compatible agents, choose their skill directory, such as `~/.agents/skills`. The installer copies one self-contained folder and refuses to overwrite an existing installation. Reload skill discovery or start a new agent session, then ask it to use **SourcePack**. Cross-host behavior beyond Codex has not been validated.
 
-Install extraction dependencies separately:
+Install extraction dependencies separately; document runtimes remain optional:
 
 - **Web:** [Summarize](https://github.com/steipete/summarize), tested with `@steipete/summarize@0.22.0` and Node >=24.
 - **YouTube:** [yt-dlp](https://github.com/yt-dlp/yt-dlp), [ffmpeg/ffprobe](https://ffmpeg.org/), plus Summarize as the first transcript candidate.
 - **Local UTF-8 text:** no external Python packages or media tools.
+- **PDF/DOCX/PPTX/XLSX:** isolated MarkItDown profile; exact qualified build and setup in [setup](skills/sourcepack/references/setup.md).
+- **Docling layout:** experimental adapter is disabled pending offline model qualification. OCR is disabled.
 
 The skill performs a readiness check and reports missing capabilities. It does not auto-install software, buy credits, download models or change provider settings. [Setup and executable overrides](skills/sourcepack/references/setup.md).
 
@@ -40,17 +42,17 @@ The skill performs a readiness check and reports missing capabilities. It does n
 5. Explain findings with timestamps, source locations and explicit gaps.
 6. Save observations so a later search can retrieve the supporting text or original frame.
 
-If interrupted, repeat the preparation command and resume pending inspection tickets. A downloaded source is not automatically marked understood. The host records what it actually inspected; uninspected material remains a gap.
+If interrupted, repeat the preparation command and resume pending reading packets. If video acquisition fails, available transcript evidence remains readable; retry only the failed stage. A downloaded source is not automatically marked understood. The host records what it actually inspected; uninspected material remains a gap.
 
 ## Direct helper use
 
 ```sh
 python3 skills/sourcepack/scripts/sourcepack.py doctor
 python3 skills/sourcepack/scripts/sourcepack.py prepare ./notes.md --out /tmp/my-sourcepack-run
-python3 skills/sourcepack/scripts/sourcepack.py next /tmp/my-sourcepack-run
-# The host reads/opens the supplied evidence, fills the returned result_template,
-# and saves result.json before submitting it.
-python3 skills/sourcepack/scripts/sourcepack.py submit /tmp/my-sourcepack-run ./result.json
+python3 skills/sourcepack/scripts/sourcepack.py read /tmp/my-sourcepack-run
+# The host reads/opens the packet and saves packet_id, inspected_ids, observations
+# and gaps in annotations.json. See SKILL.md for the small annotation format.
+python3 skills/sourcepack/scripts/sourcepack.py record /tmp/my-sourcepack-run ./annotations.json
 python3 skills/sourcepack/scripts/sourcepack.py query /tmp/my-sourcepack-run "retry limit"
 python3 skills/sourcepack/scripts/sourcepack.py export /tmp/my-sourcepack-run /tmp/my-sourcepack-export
 ```
@@ -61,11 +63,11 @@ Choose a new output directory for each source or refreshed snapshot. Reusing the
 
 This is an early, skill-first release. YouTube supports one finite watch URL up to one hour, English captions, and a video-only MP4 rendition up to 720p/256 MiB. Clips remain under the core's five-minute/64 MiB bounds. Keyframe alignment can cause a clip to exceed those bounds; that fails explicitly. Frame inspection is sparse and can miss brief events.
 
-Public webpages use saved extractor output rather than original HTML snapshots. Local text supports `.txt`, `.md`, `.html`, `.htm`, `.vtt` and `.srt`. PDFs, office files, arbitrary media, playlists, authenticated pages and automatic ASR are not integrated. The host can use existing tools for those sources, but the helper does not claim support.
+Public webpages use saved extractor output rather than original HTML snapshots. Local text supports `.txt`, `.md`, `.html`, `.htm`, `.vtt` and `.srt`. The optional basic profile converts local PDF, DOCX, PPTX and XLSX with honest derived locations. It does not establish visual coverage; image-only PDFs need separate visual inspection. Arbitrary media, playlists, authenticated pages and automatic ASR are not integrated. The host can use existing tools for those sources, but the helper does not claim support.
 
 Search is lexical. Observations and image inspection are model self-reports. The core checks ownership and citation structure; it cannot certify semantic truth. SourcePack controls no host model billing. Local storage does not imply local inference. Source URL checks are not a network sandbox.
 
-Runs contain private evidence and possibly sensitive URLs/paths. Keep them outside this repository. Exported source packs also remain private by default; retain the run for acquisition originals and logs.
+Runs contain private evidence and possibly sensitive URLs/paths. Keep them outside this repository. Exported source packs also remain private by default; default exports include acquisition originals; retain the run for resume and diagnostic logs.
 
 ## Why this exists
 
@@ -81,6 +83,6 @@ Media tests need ffmpeg/ffprobe; otherwise they skip explicitly. Tests use local
 
 [Reuse decisions](docs/REUSE.md) · [Design](docs/DESIGN.md) · [Contracts](skills/sourcepack/references/contracts.md) · [MIT licence](LICENSE)
 
-## Planned next iteration
+## Evaluation and design
 
-[Seven-repository review and implementation plan](docs/planning/README.md): simpler video inspection, partial recovery, and optional MarkItDown/Docling document adapters. These are planned changes; the supported formats and limits above describe the current runtime.
+[Seven-repository design](docs/planning/README.md) guided the 0.3 implementation. [Evaluation protocol](docs/acceptance/one-skill-protocol.md) and bundled eval cases distinguish helper tests, live conversion, answer quality and automatic triggering. See [verification](docs/VERIFICATION.md) for actual outcomes and remaining qualification limits.

docs/VERIFICATION.md

@@ -48,3 +48,7 @@ The installer copied the skill into Codex discovery with private, per-install ex
 - Public rights to captured media. Tests and reports in this repo contain no captured user evidence.
 
 All analysis evidence, local machine paths and raw receipts remain outside the published repository. No model downloads or paid extraction were used.
+
+## 0.3 implementation and evaluation
+
+See [2026-09-19 qualification results](acceptance/2026-09-19-results.md) for the current regression, real document-conversion, paired answer and fresh-context retrieval evidence. The paired pilot found equal selected-answer quality and higher SourcePack elapsed time; no general accuracy or speed advantage is claimed. Docling remains disabled pending offline model qualification.

docs/acceptance/2026-09-19-results.md

@@ -0,0 +1,73 @@
+# SourcePack 0.3 qualification results
+
+**Decision: continue as a small experimental skill; simplify its interaction overhead before promoting it as a default workflow.** This pass supports reusable acquisition, evidence storage and recovery. It does not show better answers or faster analysis than direct tools.
+
+## What was built
+
+- Timed Summarize extraction with lossless grouped captions and original cue mappings.
+- Independent acquisition receipts and usable partial runs when video fails.
+- Read/record packets with content integrity and existing lease/citation checks.
+- Bounded overview/scene/targeted frames, decoded source timestamps and contact sheets.
+- Optional, isolated MarkItDown conversion for local PDF/DOCX/PPTX/XLSX.
+- Portable private packs containing answers, source text, original artifacts, visual navigation, observations and gaps.
+- A disabled experimental Docling adapter: model completeness is not yet qualified, and OCR is disabled.
+- Frozen Agent Skills-style case files, comparison directories, strict grading aggregation and trigger queries.
+
+## Verification
+
+| Check | Observed result | Limit |
+|---|---|---|
+| Default regression suite | 86 tests: 84 passed, 2 optional live tests skipped | Local Python 3.14; CI separately reports other versions |
+| Explicit real MarkItDown tests | Both live tests passed: four document formats plus scanned-PDF boundary | Authored small fixtures; not a broad document benchmark |
+| Skill format validator | Passed | Format validation does not establish usefulness |
+| Original long-caption replay | 1,122 cues → 36 blocks → 5 text packets | Old workflow needed at least 141 packets; mappings preserve original text/timing |
+| Failed-media recovery | Available transcript remains readable; failed video retry does not reacquire successful metadata/transcript | Injected failures; live YouTube can fail differently |
+| Independent code review | Five Important findings reproduced and fixed with regressions | No Critical findings; no claim of exhaustive hostile-document resistance |
+| Optional layout models | Missing/unqualified artifact set fails before converter startup | No live Docling layout/OCR qualification or separate model downloads |
+
+The five review repairs cover skipped-stage upgrades, packet presentation integrity, portable Markdown citation rewriting, contact-sheet failure fallback, and false-positive model readiness. A singular/plural observation-search miss found in the live follow-up was also repaired and checked against the saved run. Scene-selected frames now avoid a misleading requested-point label; actual decoded PTS remains authoritative.
+
+MarkItDown qualification used commit `945314a45ddbe02935f2fd287b797dc0ba4a01e4`, version 0.1.8b3, selected format extras and an isolated Python 3.12 environment. Its Magika dependency includes a file-type classifier; no separate Docling/ASR/LLM weights were fetched. The worker denies Python socket connections and disables plugins/LLM clients, but is not a hardened OS sandbox.
+
+## Paired answer-quality pilot
+
+The protocol follows [Agent Skills evaluation guidance](https://agentskills.io/skill-creation/evaluating-skills). This is a small development pilot with explicit invocation, not a general benchmark, blind study, adoption test or automatic-trigger test.
+
+Both conditions inherited the same Codex host/model configuration and received the same raw evidence. The exact model identifier and token/cost telemetry were unavailable. Two small tasks shared one fresh context per condition; the unfamiliar video and restart tests used separate fresh contexts. The parent checked selected answers against sources without blinded condition labels.
+
+| Case | Direct tools | SourcePack | Finding |
+|---|---:|---:|---|
+| Notes: roles, requirements, claim attribution, malicious quoted content, retained evidence | 5/5 selected assertions | 5/5 | Equal |
+| Slides: manual review, approved handoff, correct slide locations, honest visual limits | 4/4 | 4/4 | Equal |
+| Unfamiliar video: workflow/setup, shown setting, recommendation/outcome distinction, version uncertainty | 4/4 selected questions | 4/4 | Equal; not every sentence independently graded |
+| Fresh-context notification-setting follow-up | Correct frame recovered | Correct frame recovered | Both worked; SourcePack initially needed plural search |
+
+The unfamiliar source was Anthropic's [Mastering Claude Code in 30 minutes](https://www.youtube.com/watch?v=6eBSHbLKuN0), 1,687 seconds. Acquisition succeeded with 24 grouped transcript sections, 95 overview frames and eight sheets. Both analyses read all available transcript text and viewed eight sheets; direct tools opened four native frames and SourcePack opened five. The rest were not claimed fully inspected. SourcePack recorded 90 native-frame gaps and 22 structural answer citations.
+
+The checked visual example is `/config` → “Turn on notifications.” It is a slide recommendation, not proof that the option was enabled. The visible `@anthropic-ai/claude-cli@0.2.114` belongs to example npm output and does not establish the running application's version throughout the recording. These distinctions were correct in both answers.
+
+## Time, interventions and cost
+
+| Measurement | Direct tools | SourcePack |
+|---|---:|---:|
+| Held-out analysis + evidence saving, acquisition excluded | 120.542 s | 447.287 s |
+| Fresh-context follow-up | 33.05 s | 41.50 s |
+| Reported user interventions | 0 | 0 |
+| Tokens / monetary cost | Unknown | Unknown |
+
+SourcePack was about **3.7× slower** on this video trial. It produced a larger standardized pack and recorded 19 inspection packets; direct tools retained a smaller bespoke evidence set. Filesystem latency and tool-call overhead were observed, but their individual causal contributions were not isolated. This is a reason to reduce ceremony, not a speed claim. The two small baseline tasks lacked complete elapsed timers, so no speed comparison is reported for them.
+
+The singular `notification` query initially missed the plural observation. The evaluated agent recovered using `notifications`; the release subsequently gained a labelled prefix fallback. A replay against the same saved run now retrieves the frame with the singular query. The original measured result remains unchanged.
+
+## What remains unproved
+
+- Repeated-run reliability and the full six-case agent-evaluation set. The frozen runner reports unrun cases as pending, never passed.
+- Automatic selection accuracy. Twenty positive/negative trigger queries are shipped with train/validation splits, but the 60-invocation sweep has not run.
+- General PDF/Office fidelity, complex table reading, layout/OCR, and performance across hosts.
+- Lower total cost, better factual reasoning, continuous video understanding or user adoption.
+
+The evaluation snapshot preceded the final recovery/export fixes; those changes have focused regressions and final copied-install smoke verification. Private traces, source bytes and captured frames stay outside Git. Public results contain no user evidence pack.
+
+## Recommendation
+
+Keep the project and the one-skill architecture. Use the structured workflow when repeatable acquisition, inspectable citations or later retrieval matter. Direct tools remain a reasonable choice for a one-off answer. Do not grow a platform or add more orchestrators. The next improvement should reduce inspection/recording steps while preserving explicit image-inspection and citation boundaries; measure it against this saved baseline before claiming a better experience.