[Refactor](feat) Modular pipeline architecture - #32
Merged
Conversation
compile-fix first, then IR patches Phase 1: build new LLVM → compile TA → AI fix compile errors Phase 2: OP analysis → generate IR patches → rebuild LLVM → build TA → test+fix loop with embedded IR retry on failure
…figuration Make test execution flexible so different test suites (pytest ut, unittest, custom scripts) can be composed via environment variables: Config (config.py): - Add TA_TEST_COMMAND env var → config.test_command - Add TA_EXTRA_TEST_DIRS env var → appends to config.test_dirs - Add _resolve_test_dirs() helper: primary dir + extras, dedup, comma/space - Preserve MAX_JOBS→BUILD_PROCS / NUM_PROCS→TEST_PROCS backward compat Test runner (test.py): - Rename run_pytest → run_tests as public entry point - Default mode: always runs pytest ut (TA_TEST_DIR + TA_EXTRA_TEST_DIRS) - Custom mode: if TA_TEST_COMMAND is set, runs it AFTER pytest ut via bash -c, with stdout/stderr captured to test-output.log - Tests pass only if ALL suites pass; failures merged into fix_errors - _run_custom_test: parses --junitxml=... from command for pass/fail counts - OOM detection: also scans test-output.log for custom test failures CLI (main.py): - Add --extra-test-dirs and --test-command CLI arguments - Document all env vars in module docstring Docs (workflow.md): - Add complete environment variable reference table - Add CLI arguments reference
…racking Default pytest UT runs first, then extra test dirs one by one, then custom test command last. Each suite writes its own JUnit XML and result JSON so failures don't clobber each other. - run_tests: sequential orchestration (primary → extras → custom) - _run_pytest: accept explicit test_dirs + label for unique log names - _run_custom_test: use per-suite result file (test-result-custom.json) - detect_oom_in_tests: scan all pytest-junit-*.xml files - _collect_test_error_logs: gather all per-suite logs for AI context
Run test_add.py with its own fix loop before any other tests. If the pre-test fails it follows the same OOM→AI fix→rebuild→retry pattern as the main loop, preventing wasted time on a full suite that would fail due to a fundamental build issue caught early. - _run_pretest_and_fix: single-file fix loop (test_procs=1) - Called at the start of test_and_fix_loop before the main loop - Skips gracefully if the pre-test file doesn't exist
test_add.py smoke check with its own fix loop runs inside the main retry loop, not just once at the start. Every retry (including after AI fix + rebuild) verifies the pre-test passes before running the full suite.
Add AI-generated PR description for the upstream sync workflow and fix redundant type prefix in AI-authored commit subjects. After the modular pipeline refactor, several capabilities were lost or degraded. This round restores and improves the PR description generation and commit message formatting. - Invokes AI in "report" mode with full sync context (step summaries, fix counts, commit lists, per-step details) - Generates structured PR body with Summary, Background, Changes, Impac - Falls back to basic template when AI is unavailable - Rewrite report mode instructions to output GitHub PR format (English) - Remove type prefix requirement from commit subject — AI writes plain description, workflow wraps as `[Sync](fix) ...` - Pass config to finalize() for AI skip check - PR descriptions now provide reviewers with comprehensive context - Commit subjects no longer have redundant "fix:" prefix - No behavioral change when SKIP_AI_ANALYSIS is set
AI test fixes were restricted to third_party/ascend/ only, but upstream API changes in python/triton/extension and libentry.py can also cause test failures that need direct fixes. - fix.py: replace single ALLOWED_PREFIX with _ALLOWED_FIX_PREFIXES list - fix.py: update rejection message to show all allowed paths - prompt.md: update fix mode and self-review sections with new paths - Allowed paths: third_party/ascend/, python/triton/extension, python/triton/runtime/libentry.py
TecJesh
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactored the Triton-Ascend upstream sync workflow from a monolithic 5000+ line CrewAI Flow into a modular pipeline architecture, then restored and enhanced capabilities lost during the refactor.
Why
The original
flow.pywas a single 5000+ line file mixing orchestration, business logic, and I/O — hard to maintain, test, and debug. This refactor splits it into 12 independent pipeline step modules, each with a single responsibility, and replaces mutable state with an immutableWorkflowContextfor better testability and traceability.Multiple rounds of behavioral alignment fixed regressions against pre-refactor behavior and added net-new capabilities.
Changes
Architecture (c6709a1)
flow.py: 5189 → 234 lines, pure orchestrationpipeline/(~100-400 lines each): prepare, detect, plan, merge, resolve, build, test, fix, ir_patch, commit, finalize, push_prutils/config.py,utils/context.py,utils/logging.py,utils/tracker.py,utils/submodule.pyWorkflowContextwithcopy_with()replaces mutableBaseModelstateIR Patch Pipeline (LLVM version changes)
git grep+git diffto automatically compare.tddefinitions between baseline and target LLVM for affected OPsTest Infrastructure
TA_TEST_DIR+TA_EXTRA_TEST_DIRS+TA_TEST_COMMANDtest_add.py) with its own complete fix loop before every test retryAI Integration
commit_message.txt(priority: commit_message.txt → step_summary.md → fallback)third_party/ascend/,python/triton/extension/,python/triton/runtime/libentry.pyPush/PR Flow
TA_FORK_OWNER→gh-proxy.test.osinfra.cn)gh pr createfor proxy compatibilityBehavioral Parity Fixes
:(top)pathspec,source_touchingcount_collect_test_error_logsgathering all per-suite logs for AI contextSKIP_LLVM_REBUILDenv var forwardingcleanup_temp_filesbefore every commit path[Sync](feat) Merge upstream commits for step-N (start..end, N commits)