Skip to content

refactor: complete project restructure #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hipudding wants to merge 3 commits into
base: main
Choose a base branch
from
Open

refactor: complete project restructure #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6f6a1f7
refactor: complete project restructure — modular pipeline with retry/…
hipudding Jul 21, 2026
3230d8a
chore: remove dead code and format all python files
hipudding Jul 21, 2026
c8a0645
refactor: split prompt into build_fix/conflict/test_fix, replace fix.…
hipudding Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ src/TA_main2main_workflow/workspace/
src/TA_main2main_workflow/output/
src/TA_main2main_workflow/__pycache__/
*.pyc
.vscode/
304 changes: 154 additions & 150 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,183 +1,187 @@
# TA Main2Main Upgrade Flow
# TA Main2Main Workflow

Automate triton-ascend's main2main upgrade against upstream Triton.
Automate triton-ascend's upstream sync against Triton main branch.

Each time Triton's `main` advances, triton-ascend must catch up: merge
upstream changes, resolve conflicts, fix broken interfaces, build, and run
e2e tests. This project drives that whole loop:
upstream changes, resolve conflicts, fix broken interfaces, rebuild LLVM (if
needed), build triton-ascend, and run tests. This project drives that whole
loop with AI-assisted fix/retry.

- detect the commit gap between triton-ascend and upstream Triton
- create a work branch from the latest triton-ascend main
- merge the target upstream commit into the work branch
- run AI (opencode or claude) to resolve conflicts and fix build/test failures
- run pytest on Ascend NPU, retry on failure (up to 3×)
- when everything passes, optionally push a branch and open a PR
## Pipeline

Full walkthrough lives in [`docs/guide.md`](docs/guide.md); this README only
covers how to install and run.
```
prepare → detect → plan → per-step loop:
merge → [resolve] → build ⇄ fix → test ⇄ fix → commit
→ finalize → [push PR]
```

| Phase | Description |
|-------|-------------|
| prepare | Clone triton-ascend, configure `origin` / `triton-upstream` remotes, fetch, checkout base branch |
| detect | Find merge-base between ascend HEAD and upstream target, list commits to merge |
| plan | Group commits into steps by line budget, handle LLVM-hash changes as solo steps |
| merge | `git merge --no-ff` the step's end commit into the current branch |
| resolve | AI resolves merge conflicts (skipped if no conflicts) |
| build | Rebuild LLVM (with AI fix if needed), then build triton-ascend. Retries on failure |
| test | Run pytest on Ascend NPU. Retries with AI fix on failure |
| commit | `git add -A && git commit -s` with structured message |
| finalize | Generate cumulative patch and summary report |
| push PR | Push branch and create GitHub PR (opt-in) |

## Requirements

- Python 3.10–3.13
- [`opencode`](https://opencode.ai) or `claude` CLI on `$PATH` (used as the AI adapter)
- `git`, plus local checkouts of `triton` and `triton-ascend`
- For real e2e tests: a host with Ascend NPUs
- For automated PRs: [`gh`](https://cli.github.com/) logged in
- LLVM toolchain (for building triton-ascend C++ extensions)
- Python 3.10+
- `opencode` or `claude` CLI on `$PATH` (AI adapter)
- `git`, `cmake`, `ninja`, `clang`/`clang++` (for LLVM build)
- For tests: a host with Ascend NPUs
- For auto PR: `gh` CLI logged in

## Install

```bash
pip install -e .
```

This registers the `ta-kickoff` and `ta-plot` console scripts.

## Run

```bash
ta-kickoff \
--triton-ascend-path /path/to/triton-ascend \
--triton-path /path/to/triton \
[--target-commit <40-char SHA>]
```

- Both paths must be local git checkouts.
- `--target-commit` is optional; defaults to upstream triton `HEAD`.
- Each run wipes and recreates `workspace/` inside the installed package directory
(under `src/TA_main2main_workflow/` in editable installs, or
`<site-packages>/TA_main2main_workflow/` with `pip install`).
Use `TA_MAIN2MAIN_WORKSPACE` env var to override the location.

CLI flags can also be supplied via env vars: `TRITON_ASCEND_PATH`, `TRITON_PATH`,
`TRITON_TARGET_COMMIT` (CLI wins).
Registers the `ta-kickoff` console script.

### Common variations
## Quick Start

```bash
# Dry-run plumbing: skip both AI and NPU tests
SKIP_AI_ANALYSIS=true SKIP_E2E_TEST=true ta-kickoff \
--triton-ascend-path /path/to/triton-ascend --triton-path /path/to/triton

# Target a specific upstream commit
ta-kickoff \
--triton-ascend-path /path/to/triton-ascend \
--triton-path /path/to/triton \
--target-commit abc123def456

# Auto-push a branch and open a PR after a successful run
PUSH_TO_GITHUB=true GITHUB_REPO=triton-lang/triton-ascend \
ta-kickoff --triton-ascend-path ... --triton-path ...

# Custom PR title: [jeshd](sync) merge upstream triton commits
PR_AUTHOR=jeshd PR_TYPE=sync PUSH_TO_GITHUB=true \
ta-kickoff --triton-ascend-path ... --triton-path ...
```

### PR title format
# Auto-clone triton-ascend to workspace/, sync to a specific upstream commit
ta-kickoff --target-commit 99f44dd5a90c9ae30daa974704fcea0bcc4f5ba1

PR titles follow the pattern `[user](type) description`:
# Use existing local repo
ta-kickoff --triton-ascend-path /path/to/triton-ascend --target-commit abc123

```
[TA](sync) merge upstream triton commits (20240612-120000)
# Dry-run: skip AI and tests (verify merge + build only)
SKIP_AI_ANALYSIS=true SKIP_BUILD=false SKIP_E2E_TEST=true ta-kickoff --target-commit abc123
```

- `user`: from `PR_AUTHOR` env var, falls back to `git config user.name`, then `TA`
- `type`: from `PR_TYPE` env var, defaults to `sync`

### Pre-commit before PR

Before pushing and creating a PR, the flow runs:
```bash
pre-commit run --from-ref origin/main --to-ref HEAD
```

If pre-commit auto-fixes files (e.g., formatting), those changes are
automatically amended into the latest commit with `git commit --amend --no-edit`.
Temp files (`result_profiling/`, `__pycache__/`, `*.lock`, `*.pyc`) are
cleaned both before and after pre-commit to avoid accidentally committing them.

### Environment variables

| Variable | Purpose | Default |
|---|---|---|
| `TRITON_ASCEND_PATH` | triton-ascend repo path | cwd |
| `TRITON_PATH` | upstream triton repo path | cwd |
| `TRITON_TARGET_COMMIT` | target triton commit SHA | triton `HEAD` |
| `AI_BACKEND` | AI adapter: `opencode` or `claude` | auto-detect |
| `SKIP_AI_ANALYSIS` | skip AI, only run deterministic steps | `false` |
| `SKIP_BUILD` | skip the build step | `false` |
| `SKIP_E2E_TEST` | skip pytest, treat as passed | `false` |
| `PUSH_TO_GITHUB` | push & create PR after all steps pass | `false` |
| `GITHUB_REPO` | PR target, `owner/name` | `TecJesh/triton-ascend` |
| `PR_AUTHOR` | user tag in PR title, e.g. `[TA](sync) ...` | git `user.name` or `TA` |
| `PR_TYPE` | conventional commit type in PR title | `sync` |
| `LLVM_INSTALL_PREFIX` | LLVM install path for building | — |
| `CONDA_ENV` | Conda environment name | `ta-upgrade` |
| `NUM_PROCS` | parallel pytest workers | `16` |
| `AUTO_STASH` | auto-stash before sync | `false` |
| `TA_PROGRESSIVE_MERGE` | enable progressive step merge | `true` |
| `TA_LINE_BUDGET` | max source lines per step | `1000` |
| `TA_COMMIT_BUDGET` | base commit-count budget per step | `5` |
| `TA_MAIN2MAIN_WORKSPACE` | override workspace directory path | package dir |

## Outputs

Everything lands under `workspace/` inside the installed package directory.
Override with `TA_MAIN2MAIN_WORKSPACE` env var. Default location:

- **Editable install** (`pip install -e .`): `src/TA_main2main_workflow/workspace/`
- **Regular install** (`pip install .`): `<venv>/lib/.../site-packages/TA_main2main_workflow/workspace/`
## CLI Arguments

| Argument | Env Variable | Default | Description |
|----------|-------------|---------|-------------|
| `--triton-ascend-path` | `TRITON_ASCEND_PATH` | — | Local path to triton-ascend repo (auto-clone if not set) |
| `--triton-path` | `TRITON_PATH` | — | Local triton repo (offline mode, not used in remote mode) |
| `--target-commit` | `TRITON_TARGET_COMMIT` | triton-upstream/main HEAD | Upstream commit SHA to sync to |
| `--llvm-prefix` | `LLVM_INSTALL_PREFIX` | `workspace/llvm-install` | LLVM install prefix path |
| `--build-procs` | `BUILD_PROCS` | 32 | Parallel workers for ninja / cmake build |
| `--test-procs` | `TEST_PROCS` | 8 | Parallel pytest workers (`-n`) |

## Environment Variables

### Repository

| Variable | Default | Description |
|----------|---------|-------------|
| `TRITON_ASCEND_PATH` | — | Local triton-ascend path; if empty, auto-clone from URL |
| `TRITON_ASCEND_URL` | `https://github.com/triton-lang/triton-ascend.git` | Clone URL for triton-ascend |
| `TRITON_PATH` | — | Local triton repo (offline mode) |
| `TRITON_UPSTREAM_URL` | `https://github.com/triton-lang/triton.git` | Upstream Triton remote URL |
| `TRITON_TARGET_COMMIT` | — | Target upstream commit SHA |
| `TA_BASE_BRANCH` | `upstream_sync` | Base branch in triton-ascend to sync from |
| `LLVM_REPO_URL` | `https://github.com/llvm/llvm-project.git` | LLVM clone URL |
| `LLVM_INSTALL_PREFIX` | `workspace/llvm-install` | LLVM install prefix |

### AI Backend

| Variable | Default | Description |
|----------|---------|-------------|
| `AI_BACKEND` | `auto` | AI adapter: `opencode`, `claude`, or `auto` (detect) |
| `TA_AI_TIMEOUT_MINUTES` | 30 | AI call timeout in minutes |
| `TA_AI_STALE_SECONDS` | 1200 | AI stale timeout (seconds) |
| `TA_AI_MAX_STALE_RETRIES` | 3 | Max AI stale retries |

### Build / Test

| Variable | Default | Description |
|----------|---------|-------------|
| `BUILD_PROCS` | 32 | Parallel build workers (ninja `-j`, cmake, setup.py) |
| `TEST_PROCS` | 8 | Parallel pytest workers (`-n`) |

### Retry / Budget

| Variable | Default | Description |
|----------|---------|-------------|
| `TA_MAX_RETRIES` | 10 | Max retry attempts per step (build + test) |
| `TA_LINE_BUDGET` | 1000 | Max source lines per merge step |
| `TA_PROGRESSIVE_MERGE` | `true` | Enable progressive step merge |
| `TA_IR_MAX_ITERATIONS` | 3 | Max IR patch iterations |

### Skip Flags

| Variable | Default | Description |
|----------|---------|-------------|
| `TA_RESUME` | `false` | Skip steps whose output files already exist |
| `SKIP_AI_ANALYSIS` | `false` | Skip all AI calls (conflict resolution, fix) |
| `SKIP_BUILD` | `false` | Skip triton-ascend build |
| `SKIP_E2E_TEST` | `false` | Skip pytest, treat as passed |
| `SKIP_LLVM_REBUILD` | `false` | Skip LLVM rebuild |
| `SKIP_IR_PATCH` | `false` | Skip IR patch generation |
| `SKIP_BASELINE_LLVM` | `false` | Skip baseline LLVM build |

### Git / PR

| Variable | Default | Description |
|----------|---------|-------------|
| `PUSH_TO_GITHUB` | `false` | Push branch and create PR after success |
| `GITHUB_REPO` | `triton-lang/triton-ascend` | PR target `owner/name` |
| `TA_MAIN2MAIN_WORKSPACE` | `./workspace` | Override workspace directory |

## Workspace Layout

```
workspace/
├── detect.json # merge-base, target commit, changed files
├── merge_result.json # merge status, conflict info
├── merge.log # raw git merge output
├── build_result.json # build step results
├── build.log # raw build output
├── test_result.json # pytest summary
├── triton-ascend/ # auto-cloned (if no local path given)
├── llvm-project/ # auto-cloned LLVM
├── llvm-build/ # LLVM build directory (outside llvm-project)
├── llvm-install/ # LLVM install prefix
├── detect.json # merge-base, target commit, changed files
├── steps.json # step plan
├── steps/
│ └── step-N/
│ ├── merge_result.json
│ ├── build_result.json
│ ├── build.log / build.err
│ ├── llvm-cmake.log / llvm-cmake.err
│ ├── llvm-ninja.log / llvm-ninja.err
│ ├── upstream.patch
│ ├── changed_files.txt
│ └── commits.txt
├── test-logs/
│ ├── pytest.log
│ └── precommit.log
├── conflicts/ # conflict snapshots (if any)
├── fixes/ # per-fix-attempt logs
│ └── fix-<N>/
├── step-0/
│ ├── step_summary.md # AI-written summary
│ ├── step_target.patch # cumulative diff
│ └── analysis.md # fix diagnosis
├── final_summary.md # final sync summary
├── final_target.patch # cumulative patch
└── FAILURE.md # failure report (if failed)
│ └── pytest-junit.xml
├── final_summary.md
└── final_target.patch
```

## Project layout
## Project Layout

```
src/TA_main2main_workflow/
├── flow.py # CrewAI Flow: nodes, routing, retry loop
├── main.py # `ta-kickoff` / `ta-plot` CLI entrypoints
├── utils.py # filename constants + git helpers + console output
├── flow.py # Pipeline orchestrator
├── main.py # `ta-kickoff` CLI entrypoint
├── pipeline/
│ ├── prepare.py # Phase 0: workspace setup
│ ├── detect.py # Phase 1: detect upstream commits
│ ├── plan.py # Phase 2: plan merge steps
│ ├── merge.py # Phase 3: git merge
│ ├── resolve.py # Phase 3: AI conflict resolution
│ ├── build.py # Phase 3: LLVM + triton-ascend build
│ ├── test.py # Phase 3: pytest
│ ├── fix.py # Phase 3: AI fix
│ ├── commit.py # Phase 3: commit progress
│ ├── finalize.py # Phase 4: summary + patch
│ ├── pre_ci.py # pre-commit checks
│ └── push_pr.py # push + create PR
├── utils/
│ ├── config.py # TAConfig dataclass
│ ├── context.py # WorkflowContext dataclass
│ ├── git.py # run_git with built-in retry
│ ├── logging.py # TALogger
│ ├── tracker.py # timed() context manager
│ ├── errors.py # Exception types
│ └── submodule.py # AscendNPU-IR submodule helpers
├── agent/
│ ├── opencode_adapter.py # spawns `opencode run`, parses JSONL events
│ └── prompt.md # single-agent task prompt
├── reference/ # knowledge base the agent reads at runtime
│ ├── adapt-guide.md
│ ├── code-structure-guide.md
│ ├── diagnosis-guide.md
│ ├── error-pattern-examples.md
│ └── npu-oom-handling.md
└── scripts/ # deterministic helpers (no AI)
├── build_test.py
├── detect_commits.py
├── merge_upstream.py
├── plan_steps.py # step planner: splits commits by line budget
├── pre_ci_check.py
├── push_to_github.py
└── update_commit_reference.py
│ └── opencode_adapter.py # AI adapter (opencode / claude)
└── reference/ # AI knowledge base
```

For a step-by-step explanation of every node and the per-step artifacts, see
[`docs/guide.md`](docs/guide.md). For conventions and gotchas that affect code
changes to this repo itself, see [`AGENTS.md`](AGENTS.md).
8 changes: 2 additions & 6 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
[project]
name = "TA_main2main_workflow"
version = "0.1.0"
description = "TA_main2main_workflow — Triton-Ascend upstream sync using CrewAI"
description = "TA_main2main_workflow — Triton-Ascend upstream sync automation"
authors = [{ name = "Your Name", email = "you@example.com" }]
requires-python = ">=3.10,<3.14"
dependencies = [
"crewai[tools]==1.14.5"
"pydantic>=2"
]

[project.scripts]
ta-kickoff = "TA_main2main_workflow.main:kickoff"
ta-plot = "TA_main2main_workflow.main:plot"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.crewai]
type = "flow"
Loading