refactor: complete project restructure - #30
Open
hipudding wants to merge 3 commits into
Open
Conversation
…fix loops
Restructure the entire project from a monolithic CrewAI-based flow into
independent pipeline steps with immutable WorkflowContext, built-in git
retry, auto-clone, resume support, and AI-assisted fix loops.
## Architecture
- Remove CrewAI dependency (~30 transitive deps → pydantic only)
- Pipeline pattern: each step is (ctx, config) → ctx with copy_with()
- All git operations via run_git() with built-in retry on fetch/clone/push
- timed() context manager replaces start_timer/stop_timer pairs
- Python logging (TALogger) replaces all print() calls
## New pipeline steps (pipeline/)
- prepare.py — clone triton-ascend, configure remotes, fetch, checkout
- detect.py — compute merge-base, list upstream commits, resume support
- plan.py — split commits into steps by line budget, LLVM hash detection
- merge.py — git merge with conflict detection, resume support
- resolve.py — AI conflict resolution
- build.py — LLVM rebuild (checkout→patch→cmake→ninja) + triton-ascend
build with unified retry/fix loop; LLVM rebuilt each attempt
- test.py — pytest with retry/fix loop
- fix.py — AI build/test fix via opencode adapter
- commit.py — commit step progress with submodule handling
- finalize.py — cumulative patch + summary report
- pre_ci.py — pre-commit checks
- push_pr.py — push branch and create GitHub PR
## New utils (utils/)
- config.py — TAConfig dataclass (23 fields, env var + CLI priority)
- context.py — WorkflowContext immutable state carrier
- git.py — run_git() with auto-retry + quiet mode for bulk calls
- logging.py — TALogger with header/section/status/key_value/table output
- tracker.py — timed() context manager for phase-level timing
- errors.py — exception hierarchy
- submodule.py — AscendNPU-IR submodule helpers
## Key features
- Auto-clone: repos cloned to workspace/ if no local path given
- Resume: TA_RESUME=true skips steps whose output files already exist
- Remote detection: uses git remotes (origin + triton-upstream), no local
triton clone needed
- LLVM patch: auto-apply matching patch, AI generates new patch on hash change
- Parallelism: BUILD_PROCS (default 32) + TEST_PROCS (default 8) independent
- Step isolation: all per-step artifacts under workspace/steps/step-N/
## Removed
- scripts/ directory — merged into pipeline/
- CrewAI flow/scheduler — replaced by TA_Main2MainFlow orchestrator
- console.py / build_helpers.py / build_llvm.py — merged into pipeline/
- ta-plot, CONDA_ENV, AUTO_STASH, PR_AUTHOR/PR_TYPE — unused features
- Various wrapper functions (get_merge_base, has_merge_conflicts,
get_conflict_files, ensure_remote) — replaced by direct run_git() calls
Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Refactors the project from a monolithic CrewAI-based workflow into a step-oriented pipeline with centralized utilities (config/context/git/logging/tracking/errors), aiming to support resume/retry flows and cleaner orchestration.
Changes:
- Introduces
pipeline/step modules (prepare/detect/plan/merge/resolve/build/test/fix/commit/finalize/pre_ci/push_pr) and aWorkflowContext+TAConfigfor state/config flow. - Replaces the legacy flat
utils.pyandscripts/helpers with a structuredutils/package (logging, git wrapper with retry, errors, tracking, submodule helpers). - Updates CLI/packaging/docs: simplifies
main.py, updatesREADME.md, drops CrewAI dependency inpyproject.toml, and expands.gitignore.
Reviewed changes
Copilot reviewed 31 out of 34 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/TA_main2main_workflow/utils/tracker.py | Adds timed() context manager + total elapsed tracking; keeps deprecated timer APIs. |
| src/TA_main2main_workflow/utils/submodule.py | Adds AscendNPU-IR submodule commit/push helpers. |
| src/TA_main2main_workflow/utils/logging.py | Introduces TALogger and get_logger() formatting helpers. |
| src/TA_main2main_workflow/utils/git.py | Adds run_git() wrapper with retry behavior and quiet-mode logging. |
| src/TA_main2main_workflow/utils/errors.py | Adds structured exception hierarchy for workflow failures. |
| src/TA_main2main_workflow/utils/context.py | Adds WorkflowContext dataclass with immutable copy_with() updates. |
| src/TA_main2main_workflow/utils/config.py | Adds TAConfig dataclass with env parsing helpers. |
| src/TA_main2main_workflow/utils/init.py | Defines constants + re-exports for backward compatibility with old utils.py. |
| src/TA_main2main_workflow/utils.py | Removes legacy monolithic utils module. |
| src/TA_main2main_workflow/scripts/update_commit_reference.py | Removes legacy script (version.txt metadata update). |
| src/TA_main2main_workflow/scripts/pre_ci_check.py | Removes legacy pre-CI script (now in pipeline). |
| src/TA_main2main_workflow/scripts/plan_steps.py | Removes legacy step planner script (now in pipeline). |
| src/TA_main2main_workflow/scripts/merge_upstream.py | Removes legacy merge script (now in pipeline). |
| src/TA_main2main_workflow/scripts/detect_commits.py | Removes legacy detect script (now in pipeline). |
| src/TA_main2main_workflow/scripts/build_test.py | Removes legacy build+test script (now in pipeline). |
| src/TA_main2main_workflow/scripts/init.py | Keeps scripts package marker (scripts directory otherwise removed). |
| src/TA_main2main_workflow/pipeline/init.py | Adds pipeline module documentation and step signature convention. |
| src/TA_main2main_workflow/pipeline/prepare.py | Adds workspace preparation (clone/remotes/fetch/checkout) step. |
| src/TA_main2main_workflow/pipeline/detect.py | Adds detect step with resume support and workspace detect.json output. |
| src/TA_main2main_workflow/pipeline/plan.py | Adds step planning logic and per-step artifact generation. |
| src/TA_main2main_workflow/pipeline/merge.py | Adds merge step with per-step merge_result.json + resume. |
| src/TA_main2main_workflow/pipeline/resolve.py | Adds AI conflict resolution step and commits resolutions. |
| src/TA_main2main_workflow/pipeline/build.py | Adds LLVM + triton-ascend build step with retry/fix loop and patch regeneration. |
| src/TA_main2main_workflow/pipeline/test.py | Adds pytest execution with retry/fix loop and JUnit parsing. |
| src/TA_main2main_workflow/pipeline/fix.py | Adds AI fix step wrapper around opencode adapter. |
| src/TA_main2main_workflow/pipeline/commit.py | Adds step-progress commit step (includes submodule handling + cleanup). |
| src/TA_main2main_workflow/pipeline/finalize.py | Adds final summary + cumulative patch output and summary table logging. |
| src/TA_main2main_workflow/pipeline/pre_ci.py | Adds temp cleanup + conflict marker + syntax checks. |
| src/TA_main2main_workflow/pipeline/push_pr.py | Updates push/PR creation path to use new logging/pre_ci, and replaces prints with logger calls. |
| src/TA_main2main_workflow/main.py | Simplifies CLI to config-driven flow execution and new banner printing. |
| README.md | Updates documentation to reflect new pipeline architecture and env/CLI options. |
| pyproject.toml | Drops CrewAI dependency; keeps only pydantic>=2; removes ta-plot script. |
| .gitignore | Adds .vscode/ ignore entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Workspace paths (computed once at import time) | ||
| # ═══════════════════════════════════════════════════════════════════════════ | ||
|
|
||
| WORKSPACE_DIR = Path(os.getenv("TA_MAIN2MAIN_WORKSPACE", str(Path.cwd() / "workspace"))) |
Comment on lines
+41
to
+43
| def start_timer(name: str) -> None: | ||
| """Deprecated: use ``with timed(name):`` instead.""" | ||
| pass |
Comment on lines
+46
to
+48
| def stop_timer(name: str) -> float: | ||
| """Deprecated: use ``with timed(name):`` instead.""" | ||
| return 0.0 |
Comment on lines
+55
to
+58
| def warn(self, msg: str, *args, **kwargs) -> None: | ||
| # Override to use consistent prefix | ||
| super().warning(f" ⚠ {msg}", *args, **kwargs) | ||
|
|
Comment on lines
+43
to
+46
| if not test_dir_path.exists(): | ||
| log.warning(f"Test directory not found: {test_dir_path}") | ||
| return ctx.copy_with(test_passed=True) | ||
|
|
Comment on lines
+50
to
+52
| run_git(ascend_path, "add", "-A") | ||
| run_git(ascend_path, "commit", "--no-edit", "-s") | ||
| log.status(True, "Committed resolution") |
Comment on lines
+28
to
+30
| try: | ||
| run_git(ascend_path, "add", "-A") | ||
| run_git(ascend_path, "commit", "-s", "-m", msg) |
Comment on lines
+27
to
+31
| log.section("Commit AscendNPU-IR Submodule") | ||
| try: | ||
| run_git(sm, "add", "-A") | ||
| run_git(sm, "commit", "-s", "-m", commit_msg) | ||
| log.status(True, f"Committed AscendNPU-IR: {run_git(sm, 'rev-parse', 'HEAD').strip()[:12]}") |
Comment on lines
+95
to
+99
| return ctx.copy_with( | ||
| triton_ascend_path=str(ascend_path), | ||
| triton_path=str(ascend_path), # same repo, upstream via remote | ||
| target_commit=target_commit, | ||
| ascend_head=ascend_head, |
- Delete utils/errors.py (13 unused exception classes) - Remove 20+ unused fields from WorkflowContext - Remove unused config fields (ir_max_iterations, skip_ir_patch, skip_baseline_llvm) - Remove dead constants from utils/__init__.py (IR_*, EACH_STEP_*, etc.) - Remove backward-compat print_* wrappers, start_timer/stop_timer stubs - Remove unused git helpers (get_repo_head, get_modified_files) - Remove duplicate _list_commits in plan.py, reuse ctx.upstream_commits - Remove unused LLVM_BUILD_LOG constant from build.py - Remove triton_path redundancy in prepare.py - Replace print_error with log.error in push_pr.py - Remove dead accumulator fields (fix_attempts, step_details, etc.) - Format all files with ruff Signed-off-by: hipudding <huafengchun@gmail.com>
…py with ai_fix.py, update pipeline modules
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restructure the entire project from a monolithic CrewAI-based flow into independent pipeline steps with immutable WorkflowContext, built-in git retry, auto-clone, resume support, and AI-assisted fix loops.
Architecture
New pipeline steps (pipeline/)
build with unified retry/fix loop; LLVM rebuilt each attempt
New utils (utils/)
Key features
Removed