Skip to content

Commit 9d0f9e9

Browse files
mrgooniemrgoonie
authored
feat: skill references & scripts support (#9)
* feat: add skill references & scripts support - Add skill_references table (id, skill_id, title, filename, url, type, content) - Add scripts JSON column and fts_content computed column to skills table - Migration 0008: schema changes for references & scripts - Seed endpoint: upsert references with URL validation, Vectorize indexing - GitHub fetcher script: discover refs/scripts via Trees API, resumable - API: return references metadata + parsed scripts in skill detail - UI: SkillReferencesSection and SkillScriptsSection components - CLI: --include-refs and --include-scripts flags, ANSI escape sanitization - CLI: split display logic into use-display.ts for maintainability - Vectorize: index reference titles + first paragraph only (~200K vectors) - Fix: double .where() in api.skill-detail.ts favorites query * docs: update architecture and codebase docs for skill references & scripts - Add skill_references table to schema docs - Document scripts JSON + fts_content columns - Add references & scripts data flow to system architecture - Update CLI docs with --include-refs/--include-scripts flags - Update CLAUDE.md with new routes and patterns * fix: address code review findings for skill references & scripts - C1: sanitize reference content before DB insert (content-scanner) - C1: change onConflictDoNothing to onConflictDoUpdate for refs re-seed - H1: human mode shows refs/scripts by default (raw mode still opt-in) - H2: DRY — use shared fetchSkillReferences in api.skill-detail.ts - H3: add console.warn on scripts JSON parse failure - H4: pass includeRefs/includeScripts through searchAndUse fallback * docs: add comprehensive CLI reference documentation Complete reference for all 6 commands (search, use, find, publish, report, config) with options, examples, API endpoints, and error handling. * feat: add bulk recompute admin endpoint for leaderboard scores Add KV-backed checkpoint/resume system for batch recomputing composite scores across all skills. Includes: - recompute-all-skills.ts: batch processing with N+1 avoidance - recompute-state.ts: KV checkpoint management (24h TTL auto-cleanup) - api.admin.recompute.ts: GET /progress, POST /run-batch, POST /run-all - Support for resume, custom offsets, batch sizes, max-iterations guard Prevents Worker timeout via 50-batch limit per request. Admin secret auth via X-Admin-Secret header only.
1 parent b55e7b2 commit 9d0f9e9

27 files changed

Lines changed: 1874 additions & 259 deletions

‎CLAUDE.md‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ pnpm db:migrate:remote # Apply migrations to remote D1
8686
| `/api/auth/*` | auth-catchall.tsx | Better Auth |
8787
| `/api/search` | api.search.ts | None |
8888
| `/api/skills/:slug` | api.skill-detail.ts | None |
89+
| `/api/skills/:slug/references` | api.skill-references.ts | None (GET), Session/Key (POST) |
8990
| `/api/skills/:slug/rate` | api.skill-rate.ts | Session/Key |
9091
| `/api/skills/:slug/review` | api.skill-review.ts | Session/Key |
9192
| `/api/skills/:slug/favorite` | api.skill-favorite.ts | Session/Key |
@@ -99,7 +100,7 @@ pnpm db:migrate:remote # Apply migrations to remote D1
99100

100101
## Database Tables (Drizzle schema)
101102

102-
`skills`, `ratings`, `reviews`, `favorites`, `votes`, `usageStats`, `apiKeys`, `installs`
103+
`skills`, `skill_references`, `ratings`, `reviews`, `favorites`, `votes`, `usageStats`, `apiKeys`, `installs`
103104
Plus Better Auth tables: `user`, `session`, `account`, `verification`
104105

105106
## Key Patterns
@@ -113,15 +114,19 @@ Search algorithm: `./docs/search-algorithm.md`
113114

114115
**Auth**: `getSession(request, env)` for session, `requireAuth(request, env)` for redirect
115116

116-
**Search**: Query → Embed (Workers AI) → Vectorize + FTS5 parallel → RRF fusion → 8-signal Boost scoring (RRF 43%, rating 15%, stars 10%, usage 8%, success 7%, votes 7%, recency 5%, favorites 5%) → Filter → Cache (KV)
117+
**Search**: Query → Embed (Workers AI) → Vectorize + FTS5 parallel → RRF fusion → 8-signal Boost scoring (vector 50%, FTS5 21.5%, rating 3%, installs 2%, votes 0.7%, recency 1%, reviews 0.15%, favorites 0.15%) → Filter → Cache (KV)
117118

118-
**Leaderboard**: 7-signal composite scoring (rating 30%, installs 20%, stars 15%, votes 10%, success 10%, recency 10%, favorites 5%). Sort tabs (best/rating/installs/trending/newest), category filter, preview modal. Client-side interaction overlay (votes + favorites) on KV-cached data.
119+
**Leaderboard**: 7-signal composite scoring (rating 30%, installs 20%, votes 10%, rating_count 15%, recency 15%, reviews 5%, verified 5%). Sort tabs (best/rating/installs/trending/newest), category + risk_label filter, preview modal. Client-side interaction overlay (votes + favorites) on KV-cached data.
119120

120-
**Vote API**: POST `/api/skills/:slug/vote` with `{ type: 'up'|'down'|'none' }`. Rate limited 10 votes/min per user. Atomic count update via SQL subquery.
121+
**Skill Detail API**: Returns skill metadata + `references` array (title, url, type) + `scripts` array (name, description, language, url). References indexed in Vectorize for search.
121122

122-
**CLI `skillx use` resolution**: `author/skill` (two-part → DB slug `author-skill`) | `org/repo/skill` (three-part → DB slug `org-skill`, fallback register from GitHub) | `slug` (direct lookup, fallback search) | `"keywords"` (search mode)
123+
**References & Scripts**: Stored separately — `skill_references` table for external docs/links/examples (title, filename, url, type enum, content). Scripts stored as JSON in `skills.scripts` (name, description, language, url). CLI flags: `skillx use --include-refs --include-scripts` to display.
123124

124-
**Register API**: POST `/api/skills/register` with `{ owner, repo, skill_path?, scan? }`. Modes: single skill (`skill_path`), scan all SKILL.md files (`scan: true`), or backward-compat fallback (try root, then scan).
125+
**Vote API**: POST `/api/skills/:slug/vote` with `{ direction: 'up'|'down' }`. Rate limited 10 votes/min per user. Atomic count update via SQL subquery. Bidirectional: same vote direction toggles off.
126+
127+
**CLI `skillx use` resolution**: `author/skill` (two-part → DB slug `author-skill`) | `org/repo/skill` (three-part → DB slug `org-skill`, fallback register from GitHub) | `slug` (direct lookup, fallback search) | `"keywords"` (search mode). Flags: `--include-refs`, `--include-scripts` for extended content. Display logic split into `use-display.ts` module.
128+
129+
**Register & Publish APIs**: POST `/api/skills/register` with `{ owner, repo, skill_path?, scan? }`. GitHub ownership verified via collaborator check. Content scanned for security (risk_label: safe/caution/danger/unknown). CLI `skillx publish owner/repo [--path X] [--scan] [--dry-run]` — requires API key auth.
125130

126131
**Styling**: Always dark theme. Use `bg-slate-900`, `text-white`, `text-mint`, `border-mint/20`. Geist Sans/Mono fonts. Lucide icons.
127132

‎apps/web/app/components/skill-references-section.tsx‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import { FileText, BookOpen, Code, ExternalLink } from "lucide-react";
2+
3+
interface Reference {
4+
id: string;
5+
title: string;
6+
filename: string;
7+
url: string | null;
8+
type: string | null;
9+
}
10+
11+
const typeIcons: Record<string, typeof FileText> = {
12+
docs: BookOpen,
13+
api: Code,
14+
guide: FileText,
15+
};
16+
17+
export function SkillReferencesSection({ references }: { references: Reference[] }) {
18+
if (references.length === 0) return null;
19+
20+
return (
21+
<div className="mb-8">
22+
<h2 className="mb-4 text-lg font-semibold tracking-tight">
23+
References
24+
<span className="ml-2 text-sm font-normal text-sx-fg-muted">
25+
({references.length})
26+
</span>
27+
</h2>
28+
<div className="space-y-2">
29+
{references.map((ref) => {
30+
const Icon = typeIcons[ref.type ?? ""] ?? FileText;
31+
return (
32+
<div
33+
key={ref.id}
34+
className="flex items-center gap-3 rounded-lg border border-sx-border bg-sx-bg-elevated px-4 py-3 text-sm"
35+
>
36+
<Icon size={16} className="shrink-0 text-sx-fg-subtle" />
37+
<span className="min-w-0 flex-1 truncate text-sx-fg">{ref.title}</span>
38+
{ref.url && (
39+
<a
40+
href={ref.url}
41+
target="_blank"
42+
rel="noopener noreferrer"
43+
className="shrink-0 text-sx-fg-muted transition-colors hover:text-mint"
44+
>
45+
<ExternalLink size={14} />
46+
</a>
47+
)}
48+
</div>
49+
);
50+
})}
51+
</div>
52+
</div>
53+
);
54+
}

‎apps/web/app/components/skill-scripts-section.tsx‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
import { Terminal, ExternalLink } from "lucide-react";
2+
import { CommandBox } from "./command-box";
3+
4+
interface Script {
5+
name: string;
6+
command: string;
7+
url: string;
8+
}
9+
10+
/** Strip ANSI escape sequences for safe display */
11+
function sanitize(str: string): string {
12+
// eslint-disable-next-line no-control-regex
13+
return str.replace(/\x1b\[[0-9;]*[a-zA-Z]/g, "");
14+
}
15+
16+
export function SkillScriptsSection({ scripts }: { scripts: Script[] }) {
17+
if (scripts.length === 0) return null;
18+
19+
return (
20+
<div className="mb-8">
21+
<h2 className="mb-4 text-lg font-semibold tracking-tight">
22+
Scripts
23+
<span className="ml-2 text-sm font-normal text-sx-fg-muted">
24+
({scripts.length})
25+
</span>
26+
</h2>
27+
<div className="space-y-3">
28+
{scripts.map((script) => (
29+
<div key={script.name} className="space-y-1.5">
30+
<div className="flex items-center gap-2 text-sm">
31+
<Terminal size={14} className="text-sx-fg-subtle" />
32+
<span className="font-medium text-sx-fg">{sanitize(script.name)}</span>
33+
{script.url && (
34+
<a
35+
href={script.url}
36+
target="_blank"
37+
rel="noopener noreferrer"
38+
className="text-sx-fg-muted transition-colors hover:text-mint"
39+
>
40+
<ExternalLink size={12} />
41+
</a>
42+
)}
43+
</div>
44+
<CommandBox command={sanitize(script.command)} />
45+
</div>
46+
))}
47+
</div>
48+
</div>
49+
);
50+
}

‎apps/web/app/lib/db/schema.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@ export const skills = sqliteTable(
2929
upvote_count: integer("upvote_count").default(0),
3030
downvote_count: integer("downvote_count").default(0),
3131
net_votes: integer("net_votes").default(0),
32+
scripts: text("scripts"), // JSON: [{name, command, url}]
33+
fts_content: text("fts_content"), // Computed: content + ref titles (for FTS5)
3234
risk_label: text("risk_label").default("unknown"),
3335
created_at: integer("created_at", { mode: "timestamp_ms" }).notNull(),
3436
updated_at: integer("updated_at", { mode: "timestamp_ms" }).notNull(),

‎apps/web/app/lib/db/skill-detail-queries.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { eq, desc, count, avg, and, sql } from "drizzle-orm";
22
import { skills, ratings, reviews, favorites, usageStats } from "./schema";
3+
import { skillReferences } from "./skill-references-schema";
34
import type { Database } from "./index";
45

56
export async function fetchSkillBySlug(db: Database, slug: string) {
@@ -101,6 +102,21 @@ export async function fetchUsageStats(db: Database, skillId: string) {
101102
return { totalUsages: total, successRate, modelBreakdown };
102103
}
103104

105+
/** Fetch skill references (metadata only — no content for page load) */
106+
export async function fetchSkillReferences(db: Database, skillId: string) {
107+
return db
108+
.select({
109+
id: skillReferences.id,
110+
title: skillReferences.title,
111+
filename: skillReferences.filename,
112+
url: skillReferences.url,
113+
type: skillReferences.type,
114+
})
115+
.from(skillReferences)
116+
.where(eq(skillReferences.skill_id, skillId))
117+
.orderBy(skillReferences.title);
118+
}
119+
104120
/** Check user-specific data: favorite status + personal rating */
105121
export async function fetchUserSkillData(
106122
db: Database,

‎apps/web/app/lib/db/skill-references-schema.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import { sqliteTable, text, integer, index, uniqueIndex } from "drizzle-orm/sqlite-core";
2+
import { skills } from "./schema";
3+
4+
// Skill references — full markdown docs indexed in Vectorize for semantic search
5+
export const skillReferences = sqliteTable(
6+
"skill_references",
7+
{
8+
id: text("id").primaryKey(),
9+
skill_id: text("skill_id")
10+
.notNull()
11+
.references(() => skills.id, { onDelete: "cascade" }),
12+
title: text("title").notNull(),
13+
filename: text("filename").notNull(),
14+
url: text("url"),
15+
type: text("type"), // docs, api, guide, cheatsheet
16+
content: text("content").notNull(),
17+
created_at: integer("created_at", { mode: "timestamp_ms" }).notNull(),
18+
},
19+
(table) => [
20+
index("idx_skill_refs_skill").on(table.skill_id),
21+
uniqueIndex("idx_skill_refs_unique").on(table.skill_id, table.filename),
22+
]
23+
);

0 commit comments

Comments
 (0)