Improving reasoning strategy support

[liana313]

Problem Statement

Support for Cot and few-shot examples is confusing and non-intuitive. Currently there are three types of ReasoningStrategy's

• COT
• ZS_COT
• FEW_SHOT

Proposed Solution

There should be two ReasoningStratgy's, both of which can be used in isolation, or composed together:

• ReasoningStrategy.CoT - add a reasoning chain to any semantic operatr
• ReasoningStrategy.Demonstrations - adds few-shot examples. These can be user-provided, or bootstrapped, i.e. the (oracle) LLM labels num_dems rows, generates a label, and uses these as demonstrations. If composed with ReasoningStrategy.CoT, then each demonstration should have a reasoning attribute.

Use Cases

Many complex document processing tasks (eg filtering, extractions, etc) can benefit from reasoning and demonstrations. A good solution will be quite pervasive.

Alternative Solutions

As an alternative, or in conjunction, we could explore integration with existing prompt optimization frameworks. A baseline implementation would apply these strategies per (LM-based) semantic operator. A future direction may jointly optimize a chain of operators.

Checklist

• [ x] I have searched existing issues to avoid duplicates
• [ x] I have provided a clear problem statement
• [ x] I have considered alternative solutions
• [ x] I have assessed the impact and priority
• [ x] I am willing to contribute to implementation (if applicable)