bug: OpenSearch terms aggregation capped at 10,000 — silent data truncation in connector sync

[coderabbitai]

Summary

src/api/connectors.py uses an OpenSearch terms aggregation with "size": 10000 to retrieve unique document_id and filename buckets. When a workspace contains more than 10,000 unique values, results are silently truncated and some documents/filenames will be invisible to the sync logic, potentially causing missed syncs or stale data.

Location

src/api/connectors.py — approximately lines 44 and 50 (pre-existing on main)

Expected behaviour

All unique document_id / filename values should be retrieved. Either:

• Switch to a composite aggregation with pagination to handle arbitrarily large result sets, or
• At minimum, emit a warning log when the returned bucket count equals 10,000 so operators are alerted to the cap.

Actual behaviour

Silently returns at most 10,000 buckets. Workspaces with more than 10,000 unique document IDs or filenames will have the excess silently dropped.

Impact

Silent data loss during connector sync for large workspaces. No error or warning is surfaced to the operator.

References

• Identified during review of PR feat: RBAC + database migration layer #1538 (feat: RBAC + database migration layer #1538) by @edwinjosechittilappilly
• Pre-existing issue on main, not introduced by that PR

[yogichipalkatti]

interim solution approved as #1912