External vector database indexing should be exposed as connectors #167
Description
Is your feature request related to a problem? Please describe.
Currently, Pathway supports primitives for indexing vectors and performing vector search. This indexing has two main types:
- Internal indexes implemented within Pathway, such as in-memory indexes;
- External indexes, such as QDrant.
When indexing vectors into external storages, it makes sense to expose these indexes as standard output connectors, while still allowing users to use the existing indexing API.
Exposing these indexes externally provides several benefits:
- A valid use case exists where users simply read certain indexes and want to save and reuse them in another pipeline;
- It improves discoverability of the functionality for indexing data into external storage from Pathway.
Describe the solution you'd like
Introduce aliases for external indexes in Pathway that allow them to be accessed as standard output connectors (pw.io.*). Users can then perform indexing operations via these output connectors without losing access to the original API.
Describe alternatives you've considered
None.
Additional context
The existing indexing approach should remain unchanged; this change only adds aliases. Therefore, this is a non-breaking change and does not affect backward compatibility.