Apart Judge Sourcer — Part A

A repeatable CLI pipeline that sources candidate judges for an Apart Research hackathon, given a topic. A Mastra agent searches curated AI-safety lists, arXiv, and the open web, then returns a structured list with name, affiliation, a relevance signal, and best-effort public contact info.

npm install
cp .env.example .env        # add your ANTHROPIC_API_KEY
npm run source -- --topic "Digital Minds" \
  --description "Moral status, sentience, welfare of AI systems" \
  --count 20
# -> out/judges-digital-minds-YYYY-MM-DD.json  (+ .csv)

Approach

The job has four sub-tasks: find candidates, extract their details, judge topical relevance, and write a relevance signal. I gave the finding/judging control flow to a Mastra agent (Anthropic model) and kept the fetching/parsing in deterministic tools. The agent decides what to fetch and who's relevant; the tools only ever return real, fetched data — so the model can't invent a person or an email.

Sourcing order (the agent is instructed to stop once it has enough strong candidates):

1. Curated seed lists (src/sources.ts) — MATS mentors, Eleos AI, Sentience Institute, NYU Center for Mind/Ethics, Apart, FAR.AI. Each is tagged by area so it's reusable across topics.
2. arXiv (arxivSearch) — keywords → relevance-ranked papers → authors. An authored on-topic paper is a strong relevance signal.
3. DuckDuckGo (webSearch, no API key) — fallback for more names / org pages.
4. fetchPage — enrich a candidate: confirm affiliation, scrape public contact (mailto, Scholar, Twitter/X, LinkedIn).

Output is written as both JSON (source of truth, nested contact + provenance) and CSV (what a human triager actually opens / imports to Sheets/Notion).

Repeatable for the next topic

Run with a new --topic/--description. To raise quality on a new area, add a few seed URLs to src/sources.ts with the right areas tags — that's the whole extension step. This is how Part B's "AI Control" hackathon reuses the same pipeline.

Tools Used

• Languages / frameworks: TypeScript (ESM), Node 22, Mastra (@mastra/core) agent framework.
• Model: Anthropic via @ai-sdk/anthropic (default claude-sonnet-4-6 with a low 1024-token thinking budget; override with JUDGE_SOURCER_MODEL).
• Libraries: zod (schemas + structured output), cheerio (HTML/XML parsing).
• Search: DuckDuckGo keyless HTML endpoint + arXiv Atom API. No paid search key.
• Dev tooling: vitest (unit tests), tsx (run TS directly), tsc --noEmit (typecheck).

How it's structured

src/
  cli.ts                    # args (--topic/--description/--topic-file/--count), run, write, summary
  schema.ts                 # zod JudgeCandidate + result schema (doubles as model instructions)
  sources.ts                # curated seed registry, tagged by area
  html.ts                   # pure parsers: htmlToText, extractEmails, extractSocialLinks
  cache.ts                  # 7-day read-through fetch cache (reproducible + polite)
  output.ts                 # dedup/validate + JSON & CSV writers
  mastra/
    index.ts                # Mastra instance
    agents/judge-sourcer.ts # instructions + tool wiring
    tools/{curated-source,arxiv-search,web-search,fetch-page}.ts
scripts/smoke-tools.ts      # exercises the data layer live, no API key needed
test/                       # vitest unit tests for the pure functions
docs/superpowers/specs/     # design doc
outreach/                   # 2 send-ready outreach drafts

Verification

• npm test — 12 unit tests on the pure parsers (arXiv XML, DDG HTML, email/social extraction), dedup, CSV escaping, topic→source matching. All pass.
• npm run typecheck — clean.
• npx tsx scripts/smoke-tools.ts — hits arXiv, DuckDuckGo, and a real team page live (no model needed). Confirmed: arXiv returns relevant papers + authors, DDG returns results, and contact extraction pulled 4 emails + 5 socials off a real team page. This is the riskiest layer, and it's verified end-to-end against the live web.

Judgment Calls

• Agent + deterministic tools, not a free-for-all agent. Tools return only fetched data; the model orchestrates. Keeps it grounded (no hallucinated contacts) and testable.
• Best-effort public contact only. Capture an email only if it's openly listed; otherwise prefer profile links. No email guessing/inference — it's spammy and risky.
• Both JSON + CSV. Cheap to emit both; they serve different consumers.
• arXiv sorted by relevance, not recency — for sourcing, the most relevant authors matter more than the newest paper (verified: recency sort returned off-topic results).
• Curated-first. These lists are human-curated for exactly this population, so they beat open search on precision; web search is the fallback for breadth.
• Fixed/verified seed URLs live rather than trusting them — several team pages had moved (Eleos /about→/team, FAR /team→/about, NYU center → nonhumanminds.org).

What I'd Do With More Time

• Judge-reliability (noted for Part B too): multi-sample relevance scoring + calibration, and an embedding pre-filter before the LLM call to cut tokens.
• seen.json ledger so repeat runs don't re-surface the same people across hackathons.
• Per-source structured scrapers (e.g. MATS is JS-rendered — fetch its data endpoint rather than relying on the agent's web-search fallback).
• Contact verification pass (confidence scoring, dedupe emails vs. profiles).
• Notion export in addition to CSV.

Open Questions

• MATS mentors is a Next.js page; the agent falls back to web search if the static HTML has no names. A dedicated data-endpoint scraper would be more reliable.
• "Send-ready" outreach assumes a human sends the final mail; we deliberately don't wire up automated emailing in Part A.
• How aggressively should we de-dupe people who appear under multiple affiliations?

Where the 1h Cutoff Hit

Complete: full pipeline (agent, 5 tools, schema, dedup, JSON+CSV), 12 passing unit tests, typecheck clean, live-verified data layer, fixed seed URLs, 2 send-ready outreach drafts, design spec. In-progress / next step: an end-to-end agent run requires the ANTHROPIC_API_KEY (not present in my env) — with the key, npm run source -- --topic "Digital Minds" --count 20 produces the final list. After that, the immediate next step is the seen.json ledger and multi-sample relevance scoring.

Video Link https://drive.google.com/drive/folders/1ePbCKhubINA9hR6iBNh_1IFfJBtE06zV?usp=sharing