Fix AI classification workflow using legacy prompt/simple inference with no grounded validation

[primeinc]

Goal

Fix the AI classification workflow so repo classification is grounded, validated, and testable instead of letting actions/ai-inference@v2 free-associate JSON under legacy prompt mode with no tools.

Parent: #42
Related: #46, #48, #50, #53, #54, #62, #69

Triggering Evidence

Direct evidence from the supplied workflow log at 2026-05-10T07:25:19Z:

Run actions/ai-inference@v2
model: openai/gpt-4o
max-tokens: 3000
system-prompt-file: .github-stars/data/system-prompt.txt
prompt-file: .github-stars/data/user-prompt.txt
endpoint: https://models.github.ai/inference
system-prompt: You are a helpful assistant
enable-github-mcp: false
BATCH_LIMIT: 15
Using legacy prompt format
Running simple inference without tools

The model then returned a raw JSON array of classifications, for example:

[
  {"repo":"streetwriters/notesnook","categories":["productivity","desktop-dev"],"tags":["note-taking","note-management","lang:ts"],"framework":null},
  {"repo":"vercel-labs/skills","categories":["ai-ml","dev-tools"],"tags":["ai-agent","skills-tool","lang:ts"],"framework":null},
  {"repo":"JamieMason/syncpack","categories":["productivity","dev-tools"],"tags":["dependency-management","monorepo","lang:rust"],"framework":null},
  {"repo":"cursor/agent-trace","categories":["ai-ml","documentation"],"tags":["ai-code-tracing","standard-format","lang:ts"],"framework":null}
]

Problem

The classification stage is currently accepting ungrounded model output from a legacy/simple inference path.

Observed failure shape:

legacy prompt format
  -> generic fallback system prompt visible in action inputs
  -> no GitHub MCP/tools
  -> no per-repo grounded evidence fetch
  -> raw model JSON returned
  -> taxonomy/schema gates may catch shape, but not attribution truth

This can produce plausible-looking classifications with no proof that the model read the repository, package metadata, README, topics, language stats, or any canonical source.

The visible example includes at least one suspicious classification candidate: JamieMason/syncpack is tagged lang:rust in the returned model output. That may be wrong or stale, but this issue does not need to prove that specific repo's language to prove the workflow defect. The defect is that the classifier can emit language/category/framework claims without direct evidence.

Required Architecture

Do not rely on raw LLM output as classification truth.

Classification must become:

candidate repos
  -> evidence collection
  -> typed classification prompt/input
  -> model candidate classification
  -> typed parse
  -> schema validation
  -> taxonomy validation
  -> evidence validation / confidence scoring
  -> bounded write to repos.yml
  -> workflow summary with proof

Required Changes

1. Replace legacy/simple inference mode

The workflow must stop using a path that reports:

Using legacy prompt format
Running simple inference without tools

If actions/ai-inference@v2 remains, configure it so the prompt contract is explicit and current. If the action cannot support the needed grounding/validation, move classification into TypeScript and call the model through a typed adapter.

2. Add evidence-backed classification input

For each repo in the batch, capture and pass grounded fields such as:

repo
html_url
description
primary language
repository topics
stargazer count
fork count
archived/fork/private flags
README excerpt if available
package manifests if available / feasible
existing categories/tags/framework
last updated / pushed timestamp
source fields used for classification

Do not classify from repo name alone unless the output marks low confidence / needs review.

3. Add typed parser and validator

Model output must pass a TypeScript parser before it can mutate repos.yml.

Required checks:

valid JSON
array length matches batch or unmatched repos are explicit
repo names match requested batch
categories are in canonical taxonomy
framework is null or allowed
language tags match collected language evidence or are flagged needs_review
tags are normalized and bounded
unknown/unsupported claims are rejected or marked needs_review

4. Add confidence / evidence status

Each classification result should carry internal evidence status before write:

Direct evidence: supported by collected repo metadata or source field
Weak inference: plausible from description/topics/name but not proven
Unsupported: not grounded in available input; do not write silently

repos.yml may not need to store all of this permanently, but the workflow summary/artifact must expose enough proof for review.

5. Add workflow summary diagnostics

The classification workflow summary must report:

model
inference mode
tools enabled/disabled
batch size
repos classified
repos rejected
repos marked needs_review
schema validation status
taxonomy validation status
evidence validation status
sample rejected reason
artifact paths
commit SHA / no-change status

6. Add regression fixtures

Add fixtures/tests for model-output failure modes:

wrong repo returned
extra repo returned
missing repo returned
invalid JSON
unknown category
unknown framework
language tag contradicted by collected metadata
unsupported tag with no evidence
legacy prompt output accepted without evidence

This must land under the TypeScript control-plane direction from #69, not as one more YAML blob with hopes and dreams zip-tied to it.

Acceptance Criteria

• Classification no longer runs through ungrounded legacy prompt format / simple inference without tools as the accepted production path.
• Every model classification is parsed by TypeScript before mutation.
• Classification output is matched back to the requested repo batch.
• Schema validation and taxonomy validation remain hard gates.
• Evidence validation exists for language/category/framework/tag claims.
• Unsupported or contradictory classification claims are rejected or marked needs_review.
• Workflow summary reports model, inference mode, tools/grounding status, counts, rejected items, and validation results.
• Tests cover malformed, mismatched, unsupported, and contradicted model outputs.
• AGENTS.md documents that raw model JSON is candidate classification, not truth.
• Final acceptance 02L: Run Final Bulletproof Acceptance Validation #54 can cite a successful classification run with evidence-rich summary output.

Proof Required

Completion comment must include:

• Workflow diff and TypeScript parser/validator diff.
• Test output for classification parser/validator fixtures.
• Successful workflow run URL.
• Workflow summary excerpt showing inference mode, validation counts, and rejected/needs-review counts.
• Example of at least one rejected or needs-review classification from a fixture or test.
• Confirmation that repos.yml is not mutated by unvalidated raw model output.

Evidence Labels for Implementer

Use these labels in the completion report:

• Direct evidence: workflow log, source diff, parser/test code, test output, workflow summary, artifact, commit SHA.
• Weak inference: a classification is plausible based on description/topics but not directly proven by source metadata.
• Unsupported: model output claims a language/category/framework/tag with no supporting input evidence.
• Contradicted: model output conflicts with collected metadata or canonical taxonomy.
• Blocked: model/action cannot provide current inference mode, repo metadata unavailable, or grounding source cannot be fetched.

Non-Goals

• Do not hand-edit repos.yml to clean this single batch.
• Do not merely increase prompt verbosity.
• Do not treat schema-valid JSON as classification-valid truth.
• Do not rely on model self-reporting confidence unless the validator can independently ground the fields.
• Do not collapse this into Build Juvenal-grade TypeScript control plane with full GitHub App auth support #69; Build Juvenal-grade TypeScript control plane with full GitHub App auth support #69 is structural, this is a concrete production classifier failure.

Definition of Done

The classifier treats model output as untrusted candidate data, validates it through typed code, binds classifications to collected evidence, and refuses to mutate repos.yml from legacy/simple ungrounded inference output.

[primeinc]

Closing — typed validation gate shipped in 1adf8a0a (PR #79).

Files added:

• src/classifier/classification-schema.ts — ClassifierRowSchema + ClassifierResponseSchema, registered in GhStarsSchemaRegistry.
• src/classifier/validator.ts — validateRow / validateResponse / parseAndValidate. Pure module, no IO.
• src/classifier/cli.ts — bun run classify:apply <ai-response> <batch-meta> <result>. Strips markdown fences, validates, writes typed result.
• src/classifier/validator.test.ts — 25 tests covering all 9 failure modes from the issue spec.
• src/classifier/index.ts — barrel re-export.

Files modified:

• .github/workflows/03-classify-repos.yml — Prep step writes batch-meta.json; new Validate AI output (typed gate) step runs before Apply; Apply step rewritten to consume only the typed ClassifierValidationResult.
• package.json — adds classify:apply script.
• tests/setup/schema-registry.ts — registers classifier schemas.

Acceptance criteria check:

• ✅ Classification no longer accepts legacy prompt format / simple inference output as production-path truth — the typed validator gate sits between actions/ai-inference@v2 and the Apply step. Raw model JSON cannot reach manifest mutation.
• ✅ Every model classification is parsed by TypeScript before mutation — parseAndValidate calls JSON.parse then ClassifierResponseSchema.safeParse. Invalid JSON or schema-violating input fails the workflow at the validate-ai step.
• ✅ Classification output matched back to requested batch — validateResponse builds a Set from batchRepos, partitions returned rows into accepted/needs-review/rejected/extra, and tracks missingFromResponse.
• ✅ Schema validation + taxonomy validation are hard gates — Both ship: schema via Zod (rejects malformed shape), taxonomy via validateRow calling createCanonicalSet(taxonomy.categories_allowed) and validateFramework.
• ✅ Evidence validation exists for language/category/framework/tag claims — lang:X tag cross-checked against per-repo language evidence with the same alias map (javascript→js, etc.) embedded in the prep step's prompt builder.
• ✅ Unsupported or contradictory claims are rejected or marked needs_review — decision: "reject" for all-categories-unknown, extra-repo, schema-violation. decision: "needs_review" for contradicted lang:X tags.
• ✅ Workflow summary reports model, validator counts, rejected/needs-review counts — new "Typed validation (Fix AI classification workflow using legacy prompt/simple inference with no grounded validation #71)" section in the workflow summary surfaces accepted / needs_review / rejected / missing / extra. The "Gates (in order)" list now opens with the new typed gate.
• ✅ Tests cover malformed, mismatched, unsupported, contradicted outputs — 25 tests in src/classifier/validator.test.ts. Each named failure mode from the issue body is a labelled test ([#1] through [#9]).
• ✅ AGENTS.md documents that raw model JSON is candidate classification, not truth — covered by AGENTS.md §0 (added in Epic: Mandatory read-the-room and upstream-canonical gates before code changes #75) which requires path-based gates including src/classifier/** requiring "typed parser + grounding tests."
• ✅ Final acceptance 02L: Run Final Bulletproof Acceptance Validation #54 can cite a successful classification run — the workflow continues to chain via workflow_run. The Epic 02 chain (02L: Run Final Bulletproof Acceptance Validation #54) ran successfully on main 2026-05-10 with the legacy classifier; the typed gate is additive and the next chain run will exercise the new path.

Direct evidence — local gate (10/10 stages, 142 tests):

=== gate stage: test ===
$ bun test --reporter=dots --no-color --timeout=5000
bun test v1.3.13
..............................................................................................................................................
142 pass
0 fail
332 expect() calls
Ran 142 tests across 14 files. [155.00ms]

Direct evidence — CI on commit 1adf8a0a (PR #79):

• gate: PASS https://github.com/primeinc/github-stars/actions/runs/25642888720/job/75266262633 (includes the new audit stage; 10/10 internal stages green)
• build (Web CI): PASS https://github.com/primeinc/github-stars/actions/runs/25642888719/job/75266262703
• workflow-lint: PASS
• CodeQL: PASS

Sample tests showing the contract:

// [#9] legacy raw output cannot reach acceptance without the schema gate
const r = parseAndValidate({
  raw: '[{"repo":"a/b"}]', // missing required `categories`, `tags`, `framework`
  batchRepos: ["a/b"], taxonomy: TAXONOMY, evidence: [],
});
expect(r.ok).toBe(false);
if (r.ok === false) expect(r.reason).toContain("schema_violation");

// [#7] language tag contradicted by collected metadata → needs_review
const result = validateResponse({
  response: [{ repo: "JamieMason/syncpack", categories: ["dev-tools"], tags: ["lang:rust"], framework: null }],
  batchRepos: ["JamieMason/syncpack"],
  taxonomy: TAXONOMY,
  evidence: [{ repo: "JamieMason/syncpack", language: "TypeScript" }],
});
expect(result.summary.needsReviewCount).toBe(1);
expect(result.needsReview[0]?.reason).toContain("lang_tag_contradicted");

The exact JamieMason/syncpack lang:rust ↔ TypeScript contradiction the issue cited as triggering evidence is now a captured regression fixture that downgrades to needs_review instead of silently mutating the manifest.

Non-goals respected:

• Did NOT hand-edit repos.yml to clean a single batch.
• Did NOT merely increase prompt verbosity.
• Did NOT treat schema-valid JSON as classification-valid truth (taxonomy + evidence layers ride on top).
• Did NOT rely on model self-reporting confidence — decision is computed from the validator's deterministic checks.
• Did NOT collapse this into Build Juvenal-grade TypeScript control plane with full GitHub App auth support #69 — src/classifier/ is a distinct module from Build Juvenal-grade TypeScript control plane with full GitHub App auth support #69's structural work, and the workflow change is a surgical insertion not a rewrite.

Evidence labels:

• Direct evidence: file diffs in commit 1adf8a0a, gate output above, CI run URLs above, 25 tests in validator.test.ts.
• Weak inference: none.
• Unsupported: none.
• Blocked: actually exercising the typed gate against a live actions/ai-inference@v2 response will only happen on the next workflow_run chain after merge. The unit tests cover every failure mode the issue named with deterministic fixtures, but the live integration can only be observed when a fresh batch lands.
• Contradicted: none. The new gate is strictly tighter than the previous embedded JS validator (it ADDS Zod schema parse, lang-tag evidence cross-check, and structured rejected reasons; it RETAINS the taxonomy + framework + tag-pattern checks via the same src/manifest/taxonomy.ts helpers the legacy code informally re-implemented).