Improve self-consistency answer extraction and voting - #191
Open
lukeinglis wants to merge 2 commits into
Open
Conversation
Replace the default projection function with smart answer extraction that tries boxed answers, explicit answer patterns, and last-paragraph fallback before resorting to full-text matching. This makes self-consistency vote on answers rather than full reasoning text, matching the algorithm described in Wang et al., 2022. Also fix create_regex_projection_function to return plain strings (not 1-tuples) for single patterns, and add ProjectionPreset enum for convenient built-in presets (math, general, exact). Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
Address review feedback on the projection function changes: - Keep default projection simple (strip + lowercase) instead of aggressive heuristic-based answer extraction, avoiding surprising behavior changes for existing users - Move smart answer extraction to GENERAL preset, keeping it opt-in - Extract _extract_boxed helper for consistent nested-brace handling across both MATH preset and _extract_answer - Change ProjectionPreset from plain class to StrEnum for proper type safety and IDE support - Fix single-pattern regex to return plain string instead of 1-tuple - Fix missing .lower() in fallback path for consistent case handling - Add comprehensive tests for presets, _extract_boxed, _extract_answer Signed-off-by: Luke Inglis <lukeinglis21@yahoo.com>
Contributor
|
@lukeinglis please rebase to main/v1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
strip().lower()instead of just.strip(), adding case normalization while keeping behavior predictable. Smart answer extraction is available via opt-in presets.ProjectionPresetStrEnum(MATH,GENERAL,EXACT) andprojection_presetparameter toSelfConsistency.__init__()so callers can opt into smart answer extraction without writing custom functions.MATH: Extracts\boxed{}content with nested brace handlingGENERAL: Tries boxed extraction, explicit answer patterns ("Final Answer:", "Therefore..."), last paragraph fallback — per Wang et al., 2022EXACT: Legacy.strip()behavior (no lowercasing)create_regex_projection_functionnow returns a plain string (not a 1-tuple like('45',)) when given a single pattern, producing cleanresponse_countskeys_extract_boxedhelper: Consistent nested-brace handling used by both theMATHpreset and the_extract_answerfunction, eliminating behavioral divergence_extract_boxed,_extract_answer, Enum validation, and custom function priorityMotivation
Per Wang et al., 2022, self-consistency works by sampling diverse reasoning chains, extracting the final answer from each, and taking a majority vote over the extracted answers. The previous default
.strip()projection means every caller must supply their own projection function to get correct voting behavior.Rather than making the default aggressively heuristic (which would be a breaking change), this PR keeps the default simple (
strip().lower()) and provides opt-in presets for smart extraction. Users who want paper-correct behavior can useprojection_preset="general"orprojection_preset="math".Additionally,
create_regex_projection_functionreturned 1-tuples for single patterns (e.g.('45',)instead of'45'), which created awkward keys inresponse_countsthat every consumer had to unwrap.Backwards compatibility
consistency_space_projection_funcare unaffected — explicit functions still take priority over presetsstrip()→strip().lower(). Only case-sensitive comparisons would see different results.projection_preset="exact"projection_preset="general"orprojection_preset="math"Test plan
\boxed{42}→"42", nested braces\boxed{x^{2}+1}→"x^{2}+1")"45")"yes")ProjectionPresetis a properStrEnummath,general,exact)"exact"preset preserves legacy.strip()behaviorValueError_extract_boxedreturnsNonewhen no boxed expression found_extract_answerfalls back through patterns correctly