diff --git a/docs/08-ci-cd-pipeline.md b/docs/08-ci-cd-pipeline.md index 36a223a..4efe618 100644 --- a/docs/08-ci-cd-pipeline.md +++ b/docs/08-ci-cd-pipeline.md @@ -7,36 +7,46 @@ Forail Platform uses **GitHub Actions** as the public CI/CD pipeline. Each repos ## Pipeline Overview ``` -┌──────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌──────────┐ ┌─────────┐ -│ Checkout │──►│ Lint │──►│ Test │──►│ Build │──►│ Security │──►│ Release │ -└──────────┘ └────────┘ └────────┘ └────────┘ └──────────┘ └─────────┘ - GitHub ruff / pytest / docker pip-audit docker push - Actions tsc vitest build trivy ghcr.io/forail-platform +┌──────────┐ ┌────────┐ ┌────────┐ ┌───────────────────────────────┐ +│ Checkout │──►│ Lint │──►│ Test │───────►│ publish │ +└──────────┘ └────────┘ └────────┘ │ build → test in image → push │ + GitHub ruff / pytest / └───────────────────────────────┘ + Actions go vet vitest / envtest ONLY on a v* tag ``` Each repo's workflow file: **`.github/workflows/ci.yml`** ## Pipeline Stages -| Stage | What it does | Fails if... | -| -------------------- | ----------------------------------------------------------------- | --------------------------------- | -| Checkout | `actions/checkout@v4` clones the repo on the runner | Repo unreachable | -| Lint (Python) | `ruff check` in backend / assistant | Lint errors | -| Lint (Frontend) | `tsc --noEmit` + ESLint | TypeScript type errors | -| Test (Python) | `pytest` against `tests_standalone/` (no DB needed for fast path) | Any test fails | -| Test (Frontend) | `vitest run` | Any test fails | -| Build | `docker build` to produce the release image | Build error | -| Security (pip-audit) | CVE scan on Python dependencies | Critical CVE | -| Security (Trivy) | Container image scan | CRITICAL CVE | -| Release | `docker push` to `ghcr.io/forail-platform/*` | Only on `main` branch or tag push | +The jobs differ per repo. What each one actually runs: + +| Repo | Always (push + PR) | `publish` (v* tags only) | +| ------------------ | ---------------------------------------------------- | --------------------------------------------------------- | +| `forail-backend` | `ruff check` , `pyproject.toml` parse, standalone tests | build image → **run the Django functional tests inside the built image** → push | +| `forail-frontend` | `npm run lint`, `npm test`, `npm run build` | build and push image | +| `forail-operator` | `go vet`, `go build`, controller tests (envtest) | build and push image | +| `forail-assistant` | `pytest -q` | build and push image | ### Stage Conditions -| Stage | When it runs | -| -------------------- | ------------------------------------------------------- | -| Checkout, Lint, Test | Every push / PR | -| Build, Security | `main` branch builds and tag builds | -| Release | `main` branch builds and tag builds (push to `ghcr.io`) | +| Stage | When it runs | +| ------------ | ------------------------------------------------------------------ | +| Lint, Test | Every push and every PR | +| `publish` | **Only** when the ref is a tag matching `v*` — `if: startsWith(github.ref, 'refs/tags/v')` | + +> **A merge to `main` does not produce an image.** It runs lint and tests only. +> The image on `ghcr.io` changes when, and only when, someone pushes a `v*` tag. +> This is the single most common source of confusion: `main` can be many commits +> ahead of the newest published image, and that is by design. + +### Known gaps + +These are documented as they are, not as they should be: + +- **`ruff check . || true`** in the backend — lint output is informational; lint + errors do not fail the build. +- **No container image scanning** (no Trivy) and **no `pip-audit`** stage in any + repo's workflow today. --- @@ -54,17 +64,32 @@ No third-party credentials are required — everything runs in the GitHub-hosted ## Docker Images -| Image | Source | Description | -| ------------------------------------------------ | ---------------------------- | ------------------------------------- | -| `ghcr.io/forail-platform/forail-backend:latest` | `forail-backend/Dockerfile` | Django API + task engine | -| `ghcr.io/forail-platform/forail-backend:` | Same | Version-tagged (CalVer) | -| `ghcr.io/forail-platform/forail-frontend:latest` | `forail-frontend/Dockerfile` | React SPA + nginx | -| `ghcr.io/forail-platform/forail-frontend:` | Same | Version-tagged | -| `ghcr.io/forail-platform/forail-assistant:latest` | `forail-assistant/Dockerfile` | FastAPI + Ollama + ChromaDB (preview) | -| `ghcr.io/forail-platform/forail-operator:` | `forail-operator/Dockerfile` | Kubernetes operator | +| Image | Source | Description | +| --------------------------------------------------- | ----------------------------- | ------------------------------------- | +| `ghcr.io/forail-platform/forail-backend:` | `forail-backend/Dockerfile` | Django API + task engine | +| `ghcr.io/forail-platform/forail-frontend:` | `forail-frontend/Dockerfile` | React SPA + nginx | +| `ghcr.io/forail-platform/forail-operator:` | `forail-operator/Dockerfile` | Kubernetes operator | +| `ghcr.io/forail-platform/forail-assistant:` | `forail-assistant/Dockerfile` | FastAPI + Ollama + ChromaDB (preview) | + +Every image carries **exactly one tag: the version**, taken from the git tag with +the `v` stripped. There is **no `latest` tag** — nothing in the pipeline creates +one, so `docker pull …:latest` fails. Always pull an explicit version. + +Release-candidate tags (`v2026.07.2-rc1`) go through the same `publish` job and +produce a normal image (`…:2026.07.2-rc1`), which is how a build is validated +against a real cluster before a release is cut. All images are **public** — no pull secret required for `docker pull` or `helm install`. +To see what is actually published, ask the registry rather than guessing: + +```bash +TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:forail-platform/forail-backend:pull" \ + | python3 -c "import json,sys; print(json.load(sys.stdin)['token'])") +curl -s -H "Authorization: Bearer $TOKEN" \ + https://ghcr.io/v2/forail-platform/forail-backend/tags/list +``` + --- ## Versioning @@ -78,12 +103,18 @@ YYYY.MM.PATCH 2026.04.0 # April release ``` -The version is derived from the git tag on `forail-deploy`: +**Each repo carries its own tag.** There is no central tag that releases the +platform — `forail-backend`, `forail-frontend`, `forail-operator` and +`forail-assistant` are tagged individually, and only the repos that changed need +a new one. That is why published versions legitimately differ between components +(for example backend `2026.07.1` alongside frontend `2026.07.0`); the Helm chart +is what pins a working set together. ```bash +cd forail-backend git tag -a v2026.05.0 -m "Forail 2026.05.0" -git push origin v2026.05.0 -# GitHub Actions automatically: checkout → lint → test → build → security → push to ghcr.io +git push github v2026.05.0 +# GitHub Actions then: lint → test → build image → test inside the image → push to ghcr.io ``` --- @@ -102,11 +133,8 @@ ruff check forail/ DJANGO_SETTINGS_MODULE=forail.settings.development \ python -m pytest forail/main/tests/unit/ -q -# Security -pip install pip-audit && pip-audit -r requirements/requirements.txt - -# Build image -docker build -t ghcr.io/forail-platform/forail-backend:latest . +# Build image (tag it with a version — there is no :latest) +docker build -t ghcr.io/forail-platform/forail-backend:2026.05.0 . ``` ### Frontend @@ -120,27 +148,35 @@ npx tsc --noEmit # Tests npx vitest run -# Build image -docker build -t ghcr.io/forail-platform/forail-frontend:latest . +# Build image (tag it with a version — there is no :latest) +docker build -t ghcr.io/forail-platform/forail-frontend:2026.05.0 . ``` --- ## Release Process -1. Ensure all tests pass on both backend and frontend (GitHub Actions on PR/push must be green) -2. Update docs and release notes -3. Commit to `forail-deploy`: `git commit -m "chore: prepare release v2026.05.0"` -4. Tag: `git tag -a v2026.05.0 -m "Forail 2026.05.0"` -5. Push tag: `git push origin v2026.05.0` -6. GitHub Actions automatically: - - Runs lint + tests - - Builds Docker images with version tag - - Scans for vulnerabilities - - Pushes `ghcr.io/forail-platform/forail-backend:` and friends to GHCR +1. Ensure GitHub Actions is green on `main` for every repo being released +2. Bump `VERSION` in the repos that ship a version, and the Helm chart's + `version` / `appVersion` so it pins the images you are about to publish +3. Write the release notes (`forail-deploy/docs/RELEASE_NOTES_v.md`) and + the docs-site release page +4. Tag each changed repo and push the tag — this is what builds and publishes: + `git tag -a v2026.05.0 -m "Forail 2026.05.0" && git push github v2026.05.0` +5. Install the published chart and images into a clean cluster and run the full + regression before announcing anything +6. Create the GitHub Release + +### Validate before you release + +Cut an rc tag first (`v2026.05.0-rc1`). It publishes a real image through the +same job, so the release candidate can be installed into a clean cluster and put +through the full Cypress suite. Only then cut the real tag. Rc images stay on +`ghcr.io` — harmless, but do not point a chart at one. ### Watch out - **Never release without passing tests.** - **Tag format must have `v` prefix:** `v2026.05.0`, not `2026.05.0`. +- **Merging to `main` publishes nothing** — only a `v*` tag does. - **Image visibility** — when a new package is first pushed to `ghcr.io`, GitHub creates it as **private** by default. You must manually flip it to public via the Packages settings (`https://github.com/orgs/forail-platform/packages`). diff --git a/docs/10-contributing-guide.md b/docs/10-contributing-guide.md index e88b2c2..8ec3cd4 100644 --- a/docs/10-contributing-guide.md +++ b/docs/10-contributing-guide.md @@ -10,6 +10,28 @@ Git workflow, commit conventions, coding standards, and the PR process. 2. **Every change must be understood** — if you can't explain why, don't commit it 3. **Author of all commits is Krstan Vjestica** — never attribute tools as authors 4. **Review the diff before committing** — always +5. **Everything must pass before you commit** — not `vagrant validate`, not a syntax + check: the actual thing running. A change to the dev environment is proven by + destroying the VM and building it again; a change to the platform is proven by + a clean install plus the full Cypress regression + +--- + +## Development environment + +- **VirtualBox is the only supported provider.** libvirt/KVM must not be running + on the same host — one hypervisor owns AMD-V per boot, and the loser's guests + die with `Guru Meditation VERR_SVM_IN_USE` or wedge mid-boot. Keep `libvirtd` + masked; `forail-dev-cluster/scripts/up.sh` refuses to start if it is active. + Full root-cause writeup: `forail-dev-cluster/docs/TROUBLESHOOTING-vagrant.md`. +- **Bring the cluster up with `scripts/up.sh`**, not bare `vagrant up` — it goes + node by node in dependency order and recreates any node whose boot wedges. +- **Start from a clean state when you are testing.** `vagrant up` on an existing + VM only boots it; it does **not** re-provision. Mixing a freshly created node + with previously provisioned ones gives two different cluster CAs and a control + plane that never reaches quorum. Use `vagrant destroy -f` first. +- **The dev VMs collide on host ports.** `forail-backend` and `forail-deploy` + both forward `8013` and `8080`, so only one of them can run at a time. --- @@ -28,10 +50,13 @@ chore/update-dependencies # Maintenance ### Standard flow +**`main` is the integration branch.** There is no `devel` branch in any Forail +repo — branch from `main` and target `main` in the PR. + ```bash -# 1. Create branch from devel -git checkout devel -git pull origin devel +# 1. Create branch from main +git checkout main +git pull github main git checkout -b feature/my-feature # 2. Make changes, test, commit @@ -41,17 +66,24 @@ git add forail/main/models/my_model.py git commit -m "feat(models): add Policy model for governance" # 3. Push and create PR -git push origin feature/my-feature +git push github feature/my-feature +gh pr create --base main ``` ### Updating the branch ```bash -git checkout devel && git pull origin devel +git checkout main && git pull github main git checkout feature/my-feature -git rebase devel +git rebase main ``` +### Remotes + +Most repos have two: `github` (github.com/forail-platform, the canonical one, +where CI and releases run) and `origin` (the GitLab mirror). Push branches and +tags to **both** when you are done, or the mirror silently falls behind. + --- ## Commit Conventions @@ -121,7 +153,7 @@ cd forail/ui_next && npx tsc --noEmit - [ ] All tests pass - [ ] No lint errors - [ ] Commit messages follow conventions -- [ ] Branch is rebased on latest `devel` +- [ ] Branch is rebased on latest `main` - [ ] Changes are minimal and focused ### PR guidelines @@ -129,7 +161,7 @@ cd forail/ui_next && npx tsc --noEmit - **Keep PRs small** — ideally under 500 lines. Large PRs are harder to review. - **One concern per PR** — don't mix a bug fix with refactoring. - **Include tests** — new features need tests, bug fixes need a regression test. -- **Target `devel` branch** — all PRs merge into `devel`. +- **Target `main`** — all PRs merge into `main`. ### Review Checklist