Skip to content

Propose oddball task categories: data security and constrained creative writing - #70

Draft
RaapTechllc wants to merge 2 commits into
mainfrom
cursor/oddball-categories-proposal-e9a0
Draft

RaapTechllc wants to merge 2 commits into
mainfrom
cursor/oddball-categories-proposal-e9a0

Conversation

@RaapTechllc

@RaapTechllc RaapTechllc commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Docs only. Proposal for new task categories beyond the current compute-the-right-thing set, holding MiniBench's executable-oracle rule for scores, plus a gated human-voting design for quality on the creative categories.

  • Data security categories that work with today's graders: pii-redaction, secret-hygiene, injection-resistance (canary-based guardrail axis), safe-query, least-privilege, log-scrubbing.
  • Constrained creative writing categories that need one new grader (constraint_pack): constrained-verse, song-form, joke-form, lipogram, word-budget.
  • Compliance is graded; quality is a separate evidence class ("Taste") from blind pairwise human votes, Bradley-Terry with CIs, only over constraint-compliant outputs. It never enters a cabinet score.
  • Anti-gaming gates: OAuth identity with account age, hidden model names until commit, 1-in-5 gold pairs with quarantine, dwell-time minimum, per-voter rate and concentration caps, agreement weighting, minimum-evidence threshold before display, per-season item rotation, vendor exclusion, published audit counts and raw accepted votes.
  • Recommended first cut: minibench-oddball-v1, 30 items, security half first; voting opens after a private calibration round.

Four owner decisions listed at the end.

Open in Web Open in Cursor 

cursoragent and others added 2 commits September 14, 2026 17:15
…ve writing

Co-authored-by: RaapTechllc <RaapTechllc@users.noreply.github.com>
Co-authored-by: RaapTechllc <RaapTechllc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants