Add Visual Work Instruction Extractor skill - #290
Open
Bas (basvb1992) wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new submission, Visual Work Instruction Extractor, to the skills gallery. The submission defines an agent workflow plus a deterministic Python helper and accompanying contracts (region proposal format + normalized manifest schema) for turning mixed-layout/scanned PDF work instructions into a reviewable, evidence-grounded ZIP output.
Changes:
- Added agent-facing
SKILL.mddescribing an evidence-first PDF→pages/crops/manifest workflow and packaging requirements. - Added optional deterministic Python helper (
extract_work_instructions.py) to render PDFs, crop proposed regions, validate outputs, and produce a ZIP archive. - Added reference contracts: region proposal format documentation, manifest JSON Schema, and a starter manifest template; plus a human-facing
README.mdandmetadata.json.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| submissions/visual-work-instruction-extractor/SKILL.md | Defines the agent procedure and required ZIP output structure/contracts. |
| submissions/visual-work-instruction-extractor/scripts/extract_work_instructions.py | Implements helper commands for render/crop/validate/package and self-test. |
| submissions/visual-work-instruction-extractor/references/region-proposals.md | Documents the normalized region proposal JSON format used for deterministic cropping. |
| submissions/visual-work-instruction-extractor/references/output-schema.json | Defines the normalized manifest JSON Schema for extracted instructions and provenance. |
| submissions/visual-work-instruction-extractor/README.md | Human-facing overview, requirements, examples, and privacy notes for the gallery page. |
| submissions/visual-work-instruction-extractor/metadata.json | Registers the submission in the gallery catalog (platforms/tags/author/version). |
| submissions/visual-work-instruction-extractor/assets/manifest-template.json | Provides a starter manifest structure aligned to the output schema. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+220
to
+224
| for page_entry in proposal["pages"]: | ||
| require(isinstance(page_entry, dict), "Each region page must be an object") | ||
| image_path = safe_relative_path(page_entry.get("image"), "region page image") | ||
| source_path = output_dir / Path(str(image_path)) | ||
| require(source_path.is_file(), f"Page image is missing: {image_path}") |
Comment on lines
+80
to
+84
| def validate_manifest_data(manifest: Any, output_dir: Path | None = None) -> list[str]: | ||
| require(isinstance(manifest, dict), "Manifest root must be an object") | ||
| require(manifest.get("schemaVersion") == SCHEMA_VERSION, f"schemaVersion must be {SCHEMA_VERSION}") | ||
| document = manifest.get("document") | ||
| require(isinstance(document, dict), "document must be an object") |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Author
|
@microsoft-github-policy-service agree |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
submissions/visual-work-instruction-extractor/scripts/extract_work_instructions.py:128
- validate_manifest_data currently uses dict.get() for several fields that are required by references/output-schema.json (e.g., sequence, partNumbers, toolsAndMaterials, evidenceImage, photoImage, extractionMethod). If those keys are missing, the helper will still treat the manifest as valid and can package an output that does not conform to the documented schema.
require(instruction.get("sourcePage") in page_numbers, f"{field}.sourcePage does not identify a page")
validate_box(instruction.get("sourceRegion"), f"{field}.sourceRegion")
confidence = instruction.get("confidence")
require(isinstance(confidence, (int, float)) and not isinstance(confidence, bool) and 0 <= confidence <= 1,
f"{field}.confidence must be between 0 and 1")
submissions/visual-work-instruction-extractor/references/output-schema.json:54
- The relativePath schema pattern allows backslashes, which means a manifest could still validate while containing Windows-style paths (and even
..segments when\\is used as a separator). Since SKILL.md and the helper require forward-slash relative paths, the schema should reject any\\and treat both/and\\as path separators for the..check.
"pattern": "^(?!/)(?![A-Za-z]:)(?!.*(?:^|/)\\.\\.(?:/|$)).+"
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot stopped reviewing on behalf of
Bas (basvb1992) due to an error
August 17, 2026 13:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Visual Work Instruction Extractor, a specialized Agent Skill for turning scanned, flattened, or mixed-layout PDF work documents into grounded instruction steps, evidence crops, safety context, component details, and part numbers.
The skill keeps photos, diagrams, callouts, warnings, and nearby instruction text connected. Every extracted instruction retains traceable page evidence, while ambiguous or safety-sensitive results are marked for review rather than guessed.
Included
SKILL.mdREADME.mdPlatform and dependencies
pypdfium2andPillowenable helper-based rendering and croppingPrivacy and security
Validation
pdf, images, ZIPs, caches, and bytecode absent)npm run check:submissionspassed (91 submissions)npm run import:submissionspassednpm run buildpassed (390 pages)submissions/visual-work-instruction-extractor/Licensing
I agree that this contribution is made under the repository's MIT License.