Add optional index sorting when MemmapIndex detects unsorted data

[aniruddh-alt]

Feature Request: Add Optional Index Sorting When MemmapIndex Detects Unsorted Data

Description

Currently, when MemmapIndex (or related index types) detects that the index is not sorted, it does not attempt to sort it automatically.
This can be a problem for large datasets, as sorting manually requires additional scripts and repeated I/O, which is extremely time-consuming for multi-terabyte corpora.

Proposed Feature

• When loading a .bin or .memmap index, if unsorted order is detected, add an optional flag (e.g., sort_if_unsorted=True) to automatically sort the index before proceeding.
• Sorting should be done in-place if possible, or to a temporary file if memory constraints require.
• Log a warning when sorting is performed, including time taken.

Example:

index = MemmapIndex.load(path, sort_if_unsorted=True)

Motivation

• For large datasets (e.g., multiple TBs), discovering an unsorted index late in a processing pipeline is costly.
• A built-in sorting option would save users from having to rerun a separate preprocessing step.
• This is especially useful in workflows where indexes are generated externally or on distributed systems where ordering is not guaranteed.