Skip to content

Improve self-consistency answer extraction and voting - #191

Open
lukeinglis wants to merge 2 commits into
Red-Hat-AI-Innovation-Team:v0from
lukeinglis:fix/self-consistency-projection
Open

Improve self-consistency answer extraction and voting#191
lukeinglis wants to merge 2 commits into
Red-Hat-AI-Innovation-Team:v0from
lukeinglis:fix/self-consistency-projection

Conversation

@lukeinglis

@lukeinglis lukeinglis commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Safe default projection: Default now uses strip().lower() instead of just .strip(), adding case normalization while keeping behavior predictable. Smart answer extraction is available via opt-in presets.
  • Built-in presets: Add ProjectionPreset StrEnum (MATH, GENERAL, EXACT) and projection_preset parameter to SelfConsistency.__init__() so callers can opt into smart answer extraction without writing custom functions.
    • MATH: Extracts \boxed{} content with nested brace handling
    • GENERAL: Tries boxed extraction, explicit answer patterns ("Final Answer:", "Therefore..."), last paragraph fallback — per Wang et al., 2022
    • EXACT: Legacy .strip() behavior (no lowercasing)
  • Fix single-pattern regex: create_regex_projection_function now returns a plain string (not a 1-tuple like ('45',)) when given a single pattern, producing clean response_counts keys
  • Shared _extract_boxed helper: Consistent nested-brace handling used by both the MATH preset and the _extract_answer function, eliminating behavioral divergence
  • Tests: 15 new tests covering presets, _extract_boxed, _extract_answer, Enum validation, and custom function priority

Motivation

Per Wang et al., 2022, self-consistency works by sampling diverse reasoning chains, extracting the final answer from each, and taking a majority vote over the extracted answers. The previous default .strip() projection means every caller must supply their own projection function to get correct voting behavior.

Rather than making the default aggressively heuristic (which would be a breaking change), this PR keeps the default simple (strip().lower()) and provides opt-in presets for smart extraction. Users who want paper-correct behavior can use projection_preset="general" or projection_preset="math".

Additionally, create_regex_projection_function returned 1-tuples for single patterns (e.g. ('45',) instead of '45'), which created awkward keys in response_counts that every consumer had to unwrap.

Backwards compatibility

  • Callers passing a custom consistency_space_projection_func are unaffected — explicit functions still take priority over presets
  • The default behavior change is minimal: strip()strip().lower(). Only case-sensitive comparisons would see different results.
  • Callers needing exact legacy behavior can use projection_preset="exact"
  • Smart answer extraction is opt-in via projection_preset="general" or projection_preset="math"

Test plan

  • Boxed answer extraction (\boxed{42}"42", nested braces \boxed{x^{2}+1}"x^{2}+1")
  • Explicit answer pattern extraction ("Final Answer: 45" → "45")
  • Case normalization ("YES" → "yes")
  • Single regex returns string, not tuple
  • Multi regex still returns tuple for hierarchical voting
  • ProjectionPreset is a proper StrEnum
  • All presets initialize correctly (math, general, exact)
  • "exact" preset preserves legacy .strip() behavior
  • Invalid preset raises ValueError
  • Custom function takes priority over preset
  • _extract_boxed returns None when no boxed expression found
  • _extract_answer falls back through patterns correctly
  • Ruff linting passes
  • All 42 self-consistency tests pass

Replace the default projection function with smart answer extraction
that tries boxed answers, explicit answer patterns, and last-paragraph
fallback before resorting to full-text matching. This makes
self-consistency vote on answers rather than full reasoning text,
matching the algorithm described in Wang et al., 2022.

Also fix create_regex_projection_function to return plain strings
(not 1-tuples) for single patterns, and add ProjectionPreset enum
for convenient built-in presets (math, general, exact).

Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
Address review feedback on the projection function changes:

- Keep default projection simple (strip + lowercase) instead of
  aggressive heuristic-based answer extraction, avoiding surprising
  behavior changes for existing users
- Move smart answer extraction to GENERAL preset, keeping it opt-in
- Extract _extract_boxed helper for consistent nested-brace handling
  across both MATH preset and _extract_answer
- Change ProjectionPreset from plain class to StrEnum for proper
  type safety and IDE support
- Fix single-pattern regex to return plain string instead of 1-tuple
- Fix missing .lower() in fallback path for consistent case handling
- Add comprehensive tests for presets, _extract_boxed, _extract_answer

Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
@beatsmonster

Copy link
Copy Markdown
Contributor

@lukeinglis please rebase to main/v1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants