Skip to content

test: add offline tests for PyTorch adapters and enable coverage - #5

Open
aliakarma wants to merge 2 commits into
ai-vnv:mainfrom
aliakarma:main
Open

test: add offline tests for PyTorch adapters and enable coverage#5
aliakarma wants to merge 2 commits into
ai-vnv:mainfrom
aliakarma:main

Conversation

@aliakarma

Copy link
Copy Markdown

Summary

This PR adds deterministic offline tests for the three PyTorch model adapters (TransformerAdapter, AutoregressiveAdapter, and VLMAdapter) and removes their coverage exclusions.

Each adapter now achieves 100% statement and branch coverage through fast offline tests, allowing these modules to participate in normal CI coverage measurement without relying on downloaded models.

Motivation

The existing adapter coverage relied on slow integration tests or lacked comprehensive offline testing, so these modules were excluded from coverage measurement.

This PR closes that gap by introducing lightweight, deterministic tests that exercise adapter behavior, assessment logic, and loader fallback paths while requiring:

  • no network access
  • no model downloads
  • no GPU

The existing slow integration tests remain unchanged.

What Changed

  • Added offline test suite for TransformerAdapter
  • Added offline test suite for AutoregressiveAdapter
  • Added offline test suite for VLMAdapter
  • Added shared offline testing fixtures for tiny local models
  • Removed the three adapter entries from the coverage omit list

Validation

All repository quality gates were executed successfully:

  • ✅ Ruff formatting and linting
  • ✅ Mypy (--strict)
  • ✅ Codespell
  • ✅ Compatibility checks
  • ✅ Full test suite
  • ✅ Coverage threshold

Final repository coverage:

  • 97.24% overall
  • 100% statement and branch coverage for:
    • transformer.py
    • autoregressive.py
    • vlm.py

The only observed test failure is a pre-existing Windows-specific issue unrelated to this change.

Design Decisions

This PR uses a hybrid testing strategy:

  • Tiny locally constructed BERT and GPT-2 models are used where real model behavior provides meaningful integration coverage.
  • Lightweight recording stand-ins are used where constructing real multimodal artifacts would introduce unnecessary complexity or external dependencies.
  • Loader fallback behavior verifies both the final outcome and the fallback sequence.

No production code was modified.

Out of Scope

This PR intentionally does not:

  • modify production code
  • change public APIs
  • add or update dependencies
  • modify CI workflows
  • change existing slow integration tests

Follow-up

During implementation, several unrelated observations were identified but intentionally left out of scope to keep this PR focused:

  • Windows-specific registries.loader.list_available() behavior
  • Empty prompt behavior in AutoregressiveAdapter
  • Decoder-only right-padding warning from transformers
  • Possible future consolidation of the helper used to obtain a live transformers module reference

These are suitable candidates for separate follow-up issues or pull requests.

Checklist

  • Offline tests added
  • Coverage exclusions removed for the three adapter modules
  • Repository coverage remains above the required threshold
  • No production code changes
  • No breaking API changes
  • No new dependencies

Add comprehensive offline test coverage for the three PyTorch model adapters
(Transformer, Autoregressive, and VLM) and remove their coverage exclusions.

Highlights:
- Add 40 deterministic offline tests covering adapter behavior,
  assessment logic, loader fallback paths, and edge cases.
- Introduce reusable test fixtures for tiny local BERT and GPT-2 models.
- Verify loader fallback order and processor/generation call contracts.
- Remove the three adapter entries from the coverage omit list.
- Preserve existing slow integration tests while enabling CI coverage
  through fast offline tests.

Validation:
- Ruff format/check passed
- Mypy strict passed
- Codespell passed
- Compatibility checks passed
- Full test suite passed (except one pre-existing Windows-specific issue)
- Repository coverage: 97.24%
- 100% statement and branch coverage for:
  - transformer.py
  - autoregressive.py
  - vlm.py

No production code, API, dependency, or CI workflow changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant