guiskillsmining

GUI Skills Mining — explore desktop GUIs and mine reusable GUI-agent skills (skill.md) from the web. The system spans the whole flow, from driving a desktop to a distilled, validated skill library a computer-use agent can load.

┌──────────────┐  ┌────────────────────────┐  ┌──────────────┐
│ explorer     │  │ miner                  │  │ skills       │
│ drive a GUI  │→ │ log → query → pages    │→ │ pages →      │→ SKILL.md
│ emit log     │  │ Serper + crawl4ai      │  │ skills       │
└──────────────┘  └────────────────────────┘  └──────────────┘
        │                                                                              │
   emits log.json  ───────────── frozen contract ─────────────►  parse-logs      skills_index.jsonl

The front stage, explorer (drive a desktop, emit log.json), is a first-class but reserved subpackage: its interfaces and structure exist, the implementations are stubs to be ported from sibling prototypes. The back half — parse → query → search → crawl → distill — is fully working.

Why it's structured this way

• Folder per workflow step. The business flow is organized as top-level folders: explorer/, miner/, and skills/.
• Fixed default services. LLM calls go through Baidu Qianfan's OpenAI-compatible API, web search uses Serper, crawling uses crawl4ai, and skills are saved to the filesystem. Runtime env only needs QIANFAN_API_KEY and SERPER_API_KEY.
• One skill owner. skills/ owns schema, generation, storage and Claude Code SKILL.md rendering, so the format never drifts across generation / store / index.
• A frozen log contract. explorer/logging/log_writer.py is fully implemented and pinned by tests, so the mining half stays stable while exploration is ported later.

What a skill looks like

Mined skills are written as Claude Code-compatible Agent Skills: a <slug>/SKILL.md folder with minimal frontmatter (name == folder slug, description). Drop a folder into ~/.claude/skills/ or a project .claude/skills/ and Claude Code loads it. See examples/sample_skill.md:

---
name: windows11-magnifier-zoom
description: "Zoom the Windows 11 screen with Magnifier. Use when the user wants to enlarge content (app: Magnifier)."
---
# Zoom the screen with Magnifier
## When to use
...
## Steps
1. Press `Win` + `+` to start Magnifier.
...
## Verify / ## Pitfalls

The distiller's richer signals (contains_skill, os, app) stay out of the published SKILL.md — os/app are folded into the description and the full metadata is kept in skills_index.jsonl for mining provenance. The generated SKILL.md body also includes a ## Reference section pointing back to the crawled source URL.

Deduplication and review

The skill store deduplicates both online and offline:

• Online deduplication. Before writing a new <slug>/SKILL.md, SkillStore.save() compares it with the existing skills_index.jsonl. It skips exact source-URL repeats and semantically similar skills.
• Fingerprinting. Similarity first uses a stable os | canonical_app | intent key. For Microsoft Account workflows, synonyms such as reset / recover / forgotten password collapse to one intent, and app names like web portal / browser / login page are normalized to microsoft account.
• Text fallback. If fingerprints differ, the store compares normalized name + description text with a similarity threshold to catch near-duplicates.
• Offline review. After distillation, skills_review.json reports duplicate groups, missing files, overly simple skills, missing Verify sections, and weak descriptions. Review findings are warnings; they do not delete already generated files.

Install

uv venv skill --python 3.13.7
source skill/bin/activate
uv pip install -e ".[all]"     # crawl4ai
cp .env.example .env            # fill in endpoint + keys

Python ≥ 3.10. Needs QIANFAN_API_KEY and a Serper key.

Usage

See examples/run_pipeline.md. In short:

gsm parse-logs  --logs-root LOGS --output data/packaged.json
gsm gen-queries --input data/packaged.json --output data/queries.jsonl --os ubuntu
gsm run         --queries data/queries.jsonl --save-root save     # search+crawl+distill
gsm skills list --index save/skills_index.jsonl

Run gsm -h / gsm <cmd> -h for the full surface.

Project layout

src/
├── explorer/ # Step 1: drive GUI and emit log.json (reserved stubs + log writer)
├── miner/ # Step 2-3: parse guide trajectory, generate query, search and crawl
├── skills/      # Step 4: skill schema + generation + Claude Code SKILL.md store
├── llm/         # Qianfan chat helper + response text utils
├── prompts/     # query_generation, skill_generation prompt templates
├── main.py # logical flow + end-to-end runner
└── cli/         # the `gsm` command
tests/  examples/

data/ (datasets, outputs) and legacy/ (original prototype scripts) are git-ignored.

Development

uv pip install -e ".[dev]"
pytest                      # offline: LLM/crawl/search are mocked or lazily imported

License

Apache-2.0.