Skip to content

cicd-pipeline.md

Latest commit

 

History

History
139 lines (107 loc) · 5.59 KB

cicd-pipeline.md

File metadata and controls

139 lines (107 loc) · 5.59 KB

CI/CD Pipeline (as of April 19, 2025)

Kitware’s CI/CD system for the JATIC program is modular and highly reusable across repositories. It is designed to standardize workflows while allowing for project-specific overrides.

The root pipeline entrypoint is .gitlab-ci.yml, which includes most job definitions from the .gitlab-ci/ directory. These are further broken down by purpose.

Structure

  • .gitlab-ci.yml The top-level orchestrator file that defines pipeline stages, includes jobs, and enforces scheduling and skip rules. Other JATIC repositories typically include it like this:

    include:
  - project: "jatic/kitware/devel-jatic"
    ref: "main"
    file: ".gitlab-ci.yml"

  # Repo-specific job overrides or additions
  - local: .gitlab-ci/.gitlab-docs.yml
  - local: .gitlab-ci/.gitlab-test.yml
  - local: .gitlab-ci/.gitlab-container.yml

  • .gitlab-ci/ Contains shared CI job definitions:

    • .gitlab-test.yml: Unit tests, extras, notebooks, coverage
      • notebooks are run manually and tested nightly
    • .gitlab-quality.yml: Linters, type checks, Sphinx linting
    • .gitlab-docs.yml: Sphinx build and GitLab Pages deployment
    • .gitlab-security.yml: SAST, dependency scanning, secret detection
    • .gitlab-mirror.yml: GitHub mirror including Git LFS
    • .gitlab-publish.yml: PyPI publishing on tag
    • .gitlab-devel.yml: Automation and multi-repo content propagation
    • .gitlab-shared.yml: Shared setup used across jobs

  • .gitlab-ci/pipelines/ Contains child pipeline entrypoints used for:

    • Compliance Scanning: .gitlab-compliance.yml includes components for SR, TR, DR compliance, unit tests, and pipeline validation
    • Container Build/Scan: .gitlab-container.yml builds and scans Docker images using Harbor and Trivy

Pipeline Stages Overview

Stage Purpose Merge Request Required
test Run unit tests, notebook validation (manual), and coverage reporting ✅ ✅
quality Code linting (ruff), type checks (pyright), and doc linting (sphinx) ✅ ✅
docs Build and preview documentation ✅ ✅
mirror Push code and LFS objects to GitHub ❌ ❌
publish Publish to PyPI if tag matches version ❌ ❌
devel Propagate files across Kitware-managed repos ❌ ❌
security Run GitLab SAST, Dependency, and Secret detection scanners ✅ ❌
container Build Docker images and run vulnerability scans Manual
compliance Trigger DevSecOps child pipeline (.gitlab-compliance.yml) ✅ ❌

Each repo can selectively override, disable, or add jobs by defining its own .gitlab-ci/ entries alongside the shared pipeline.

Where to Start

Most developers interact with:

  • test and quality stages to validate code
  • docs to verify documentation previews
  • container when working on tools with a Docker interface

For infrastructure contributors, review .gitlab-ci.yml and child pipelines in .gitlab-ci/pipelines/ for advanced setup and compliance jobs.