fix(tests): Use provider_type instead of provider_id for vector-io test parametrization#5263
Draft
franciscojavierarceo wants to merge 3 commits intollamastack:mainfrom
Draft
Conversation
…st parametrization
Fixes vector-io integration tests in CI that were skipping for 7 out of 9 providers
due to a mismatch between test parametrization and validation logic.
## Problem
The CI workflow matrix correctly configured all 9 vector database providers
(inline::faiss, inline::sqlite-vec, inline::milvus, remote::chromadb,
remote::pgvector, remote::weaviate, remote::qdrant, remote::elasticsearch,
remote::infinispan), but tests were skipping for most providers.
When CI passed --stack-config="vector_io=remote::pgvector", the test
parametrization extracted provider_id ("pgvector") instead of provider_type
("remote::pgvector"), causing tests to skip because they check against full
provider type names.
## Solution
Changed three related sections in tests/integration/conftest.py to use
provider_type consistently:
1. **Test parametrization (line 260)**: Changed from p.provider_id to
p.provider_type so tests receive "remote::pgvector" instead of "pgvector"
2. **Validation helper (lines 416-424)**: Changed get_vector_io_provider_ids()
to return p.provider_type instead of p.provider_id to match parametrization
3. **Fallback list (lines 265-279)**: Updated local development fallback to
use full provider type names for consistency
## Impact
- **Before**: 2/9 providers tested (22% coverage)
- **After**: 9/9 providers tested (100% coverage)
- **Improvement**: +778% test coverage for vector-io integration tests
## Testing
Verified with verification script that:
- Provider objects have both provider_id (short) and provider_type (full)
- Parametrization now uses provider_type
- Validation helper now uses provider_type
- Both match and tests can run without skipping
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
meta-cla
bot
added
the
CLA Signed
…vider IDs The previous commit broke all tests by passing full provider type names (e.g., "remote::pgvector") to the router, which is keyed by short names (e.g., "pgvector"). This caused KeyError failures in all vector-io tests. Root cause: The capability check set _PROVIDERS_WITH_NATIVE_FILTERING contained full type names but was comparing against the short provider_id parameter from test parametrization, causing spurious test skips. Correct fix: - Revert parametrization to use short names (provider_id) so API routing works - Update _PROVIDERS_WITH_NATIVE_FILTERING to use short names for comparison - Keep other capability checks using provider_type (they query runtime API) This ensures tests run for all providers without skip or routing failures. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…onfig Make faiss and sqlite-vec providers conditional using ENABLE_FAISS and ENABLE_SQLITE_VEC environment variables, similar to how remote providers are configured. This allows the CI to enable only one provider at a time when running in server mode. This is part of enabling OpenAI client tests in the vector-io CI workflow, which requires server mode instead of library mode. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
Recording workflow finished with status: failure Providers: watsonx Recording attempt finished. Check the workflow run for details. Fork PR: Recordings will be committed if you have "Allow edits from maintainers" enabled. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use provider_type instead of provider_id for vector-io test parametrization.
Fixes vector-io integration tests in CI that were skipping for 7 out of 9 providers due to a mismatch between test parametrization and validation logic.
Problem
The CI workflow matrix correctly configured all 9 vector database providers (inline::faiss, inline::sqlite-vec, inline::milvus, remote::chromadb, remote::pgvector, remote::weaviate, remote::qdrant, remote::elasticsearch, remote::infinispan), but tests were skipping for most providers.
When CI passed --stack-config="vector_io=remote::pgvector", the test parametrization extracted provider_id ("pgvector") instead of provider_type ("remote::pgvector"), causing tests to skip because they check against full provider type names.
Solution
Changed three related sections in tests/integration/conftest.py to use provider_type consistently:
Test parametrization (line 260): Changed from p.provider_id to p.provider_type so tests receive "remote::pgvector" instead of "pgvector"
Validation helper (lines 416-424): Changed get_vector_io_provider_ids() to return p.provider_type instead of p.provider_id to match parametrization
Fallback list (lines 265-279): Updated local development fallback to use full provider type names for consistency
Impact
Testing
Verified with verification script that:
What does this PR do?
Test Plan