AttributeSearchStrategy: Add scalability support for large memory datasets

[csmangum]

Currently, our AttributeSearchStrategy provides good performance for moderate memory store sizes, but may encounter scalability issues with massive datasets. As our agent memory grows, we need to implement optimizations to maintain search performance.

Problem

When memory stores reach large sizes (thousands or millions of entries), the current implementation may experience:

• Increased search latency
• Higher memory consumption during searches
• Potential blocking of the main thread during complex searches

Proposed Solutions

1. Implement indexing support

• Add configurable indexing for commonly searched fields
• Support for both content and metadata indices
• Consider implementing a simple inverted index for text fields
• Add index refresh/rebuild strategies

2. Add asynchronous search capabilities

• Create an async version of the search method
• Support for search cancellation
• Progress reporting for long-running searches
• Background processing options

3. Consider batched/paginated processing

• Process large memory stores in chunks
• Add cursor-based pagination for search results

Technical Considerations

• Keep backward compatibility with current sync API
• Minimal memory overhead for indices
• Consider tiered approach (different indexing strategies for STM vs LTM)

Acceptance Criteria

• Indexing mechanism implemented and tested
• Async search API created
• Performance benchmarks showing improvements
• Documentation updated to reflect new capabilities
• Test suite expanded to cover new functionality

Priority

Medium - Important for future-proofing but not blocking current functionality