fix: add warning logs for 10k aggregation limit in sync operations

[yogichipalkatti]

Added warning logs to alert operators when OpenSearch terms aggregations
hit the 10,000 bucket limit in connector sync operations. This helps
detect potential data truncation issues in workspaces with >10k unique
document IDs, filenames, or connector file IDs.

Affected functions:

• get_synced_file_ids_for_connector(): Warns when connector_file_id,
  document_id, or filename aggregations reach the limit
• get_synced_id_to_filename_map(): Warns when document_id aggregation
  reaches the limit

This is an interim solution to surface the issue. A future enhancement
should implement composite aggregation with pagination to handle
arbitrarily large result sets without truncation.

Related: Connector sync may miss documents when workspace exceeds 10k
unique values per field

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 321cef32-fef6-471c-b094-8d9d01919360

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)

Validation error: Too big: expected number to be <=900000 at "reviews.tools.github-checks.timeout_ms"

⚙️ Configuration instructions

• Please see the configuration documentation for more information.
• You can also validate your configuration using the online YAML validator.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[richardz403]

Looks good to me.

The additional code correctly alerts and warns when connector file IDs, document IDs, or filename aggregations hit 10k items.

Ran locally to check for other conflicting issues, none were found.

[Vchen7629]

LGTM, verified that all the tests pass when running locally

[NishadA05]

This adds useful warning logs when the OpenSearch aggregations hit the 10,000 bucket limit, which makes possible truncation visible instead of silently returning incomplete sync data.

One note: this does not fully remove the 10k limit, but it is a helpful improvement for now.

[Wallgau]

Great work, please create an issue for the enhanced solution