Skip to content

fix(agents): replace Critic scoring with claim audit - #189

Merged
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/adversarial-critic-audit-0717
Jul 17, 2026
Merged

fix(agents): replace Critic scoring with claim audit#189
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/adversarial-critic-audit-0717

Conversation

@FluffyAIcode

Copy link
Copy Markdown
Owner

Summary

  • remove numeric Critic scoring and supportive-grader framing
  • require full semantic thesis reconstruction and claim-by-claim adversarial audit
  • require exact quoted claims, strongest objections, corrections, and residual uncertainty
  • distinguish epistemic honesty from literal task completion
  • explicitly audit numbers, awards, quantifiers, and claimed consequences
  • tag benchmark stages with critic_protocol=adversarial_claim_audit_v1
  • retain complete 923/923-token context with no sampling, summary, simplification, or fallback

Root cause

The prior Critic received the complete Generator response but was prompted to score it. Gemma therefore produced a stable supportive 10/10 judgment and overlooked material issues, including treating an unsolved problem as impossible, implying a Fields Medal is automatic, misstating the Clay prize in Chinese, and exaggerating downstream theorem consequences. Deterministic KV reuse correctly repeated that flawed protocol output.

Tests

  • deterministic adversarial protocol requirements and full-context metadata
  • targeted Agent GAN/report suite — 17 passed
  • full platform-neutral suite — 867 passed, 1 skipped
  • Python compile and diff checks pass

Made with Cursor

Make Gemma reconstruct and adversarially challenge the complete response claim by claim instead of producing supportive numeric scores and blanket approval.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Jul 17, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@FluffyAIcode
FluffyAIcode merged commit 57d17fd into main Jul 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant