Skip to content

[ExternalTest](feat) Add pluggable external operator repo test framework - #36

Open
TecJesh wants to merge 4 commits into
mainfrom
add_external_test_case
Open

[ExternalTest](feat) Add pluggable external operator repo test framework#36
TecJesh wants to merge 4 commits into
mainfrom
add_external_test_case

Conversation

@TecJesh

@TecJesh TecJesh commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

Add a pluggable external operator repo test framework. Users specify external
GitHub repositories and their test cases in a YAML config file. The workflow
automatically clones each repo, runs tests sequentially per repo, triggers AI
fix on failure, then proceeds to the next repo.

Core Design

Aspect Description
Config-driven Single YAML file (external_test_config.yaml), repos grouped with their test cases
Master switch enabled field (default false), three-tier priority: CLI > env var > YAML
Three modes inline (blocks main flow), standalone (non-blocking), off (skipped)
Execution order Strictly serial across repos (first-configured-first-run), per-repo parallel via pytest -n 8
AI fix AI fix + retest loop on failure, up to max_retries attempts, changes gated to the external repo directory

Changes

src/TA_main2main_workflow/
├── external_test/                              # [new]
│   ├── __init__.py
│   ├── external_test_config.yaml               # default config template
│   ├── config_loader.py                        # YAML loading + env override
│   └── runner.py                               # clone → install → pytest → fix
├── utils/
│   ├── config.py                               # [modified] +3 fields
│   └── context.py                              # [modified] +2 fields
└── flow.py                                     # [modified] external test stage integration

New Environment Variables

Variable Default Description
TA_EXTERNAL_TEST_ENABLED false Master on/off switch
TA_EXTERNAL_TEST_CONFIG Built-in default path Config file path
TA_EXTERNAL_TEST_MODE inline inline / standalone / off
TA_EXTERNAL_TEST_PROCS 8 pytest parallel workers per repo
TA_EXTERNAL_TEST_MAX_RETRIES 5 AI fix retry limit per repo
TA_EXTERNAL_TEST_TIMEOUT 7200 Per-repo test timeout (seconds)
  • run pre-commit run --all-files to ensure code style
  • unit tests added or updated (if applicable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant