copilot-workflow

Per-session AI cost tracking for GitHub Copilot CLI. Tracks token usage and USD cost per session, attributes costs to GitHub issues, and renders an HTML dashboard. Ships a Docker-backed shell environment (gru) for running autopilot sessions, board watchers, and cost tools.

What it does

src/cost-sync.py — called by the sessionEnd hook; reads token telemetry from events.jsonl and appends one cost record to ~/.copilot/cost-log.jsonl
src/cost-retrospective.py — backfill tool; scans all historical sessions and writes ~/.copilot/cost-log-historical.jsonl
src/cost-report.py — generates a text report or HTML dashboard from merged JSONL logs, grouped by GitHub issue
src/pricing.py — USD cost engine; mirrors the TypeScript pricing logic from devartifex/copilot-cost
scripts/watcher-run.sh — orchestrates multi-issue autopilot runs; each issue gets its own session and cost record
skills/ — Copilot CLI skills: issue-start, session-handoff, cost-report

Architecture

Copilot CLI session
  │
  ├─ issue-start skill    → writes ~/.copilot/session-state/<id>/issue-refs.json
  │
  ├─ [work happens]
  │
  ├─ session-handoff skill
  │
  └─ /new  ──► sessionEnd hook fires
                  │
                  └─ cost-sync.py
                       ├─ reads events.jsonl (session.shutdown event)
                       ├─ computes USD cost via pricing.py
                       └─ appends to ~/.copilot/cost-log.jsonl

Morning:
  python3 src/cost-report.py --format html   →  docs/cost-dashboard.html
  python3 src/cost-report.py                 →  text report to stdout

The `gru` environment

gru is a source-only, Docker-backed shell environment — modelled on docker-yocto-env.

source ./gru            # build/pull image if needed, register commands
source ./gru --rebuild  # force an image rebuild

Sourcing prints "Welcome to the Gru's Lab" and registers the plugin commands: copilot, gh-watch, md-watch, cost, data.

It must be sourced, not executed — it defines shell functions in your current shell.

File layout

gru                     # entry point (sourced). Sets SCRIPT_DIR, prints banner,
                        #   loads core/, runs _initialize_environment, lists commands.
core/
  config.sh             # all CW_* variables, PROJECT_TOP, GH_TOKEN/GH_HOST bridge
  env_core.sh           # sources common/config/lib/plugin_loader; _initialize_environment
  common.sh             # command_exists, print_*, _cw_quote, gh/copilot bootstrap snippets
  plugin_loader.sh      # register_plugin_command, load_plugins, show_plugin_commands
lib/
  docker_utils.sh       # _ensure_cw_image/_volumes/_dirs, _seed_cw_data,
                        #   _run_cw_docker, _cw_dock, _cw_dock_cmd, _cw_dock_bg,
                        #   _logui_start/stop/status (shared dashboard helpers)
plugins/
  copilot.sh            # copilot {shell|run}
  gh-watch.sh           # gh-watch [DIR] {run|start|stop|status}  (GitHub project board)
  md-watch.sh           # md-watch {run|start|stop|status} <board.md> (Obsidian Kanban)
  cost.sh               # cost {report|link|dashboard}
  data.sh               # data {update|preview}
docker/
  Dockerfile            # the gru:local image
  entrypoint.sh         # used by the autopilot *service* container (clones + runs)
  hooks.json            # sessionEnd cost hook installed into the image
  scripts/
    watch-log-ui.py     # dashboard server (pipeline board + live log stream)

The non-env directories (src/, scripts/, data/, prompts/, skills/, tests/) are the actual Python/shell tooling that the plugins invoke inside the container.

Boot sequence

gru resolves SCRIPT_DIR (the repo root) and prints the banner.
core/env_core.sh sources: common.sh → config.sh → lib/docker_utils.sh → plugin_loader.sh.
_initialize_environment:
- _ensure_cw_image — builds gru:local from docker/Dockerfile if missing (or FORCE_BUILD=true);
- _ensure_cw_volumes — creates named volumes gru-data and gru-logs;
- _ensure_cw_dirs — mkdir -p data/ docs/ on the host;
- _seed_cw_data — copies committed data/*.jsonl + attributions.db into the data volume only if absent;
- load_plugins — sources every plugins/*.sh and calls each <name>_init.
show_plugin_commands prints the registered commands.

Key configuration (`core/config.sh`)

Variable	Meaning	Default
`CW_IMAGE`	Image tag	`gru:local`
`CW_DATA_VOLUME` / `CW_LOGS_VOLUME`	Named volumes	`gru-data` / `gru-logs`
`PROJECT_TOP`	Repo root. Derived from `SCRIPT_DIR`, never `git rev-parse` — correct when used as submodule.	`${SCRIPT_DIR}`
`GH_HOST`	GitHub Enterprise host	`github.com`
`GH_TOKEN`	Auth token. If unset, auto-derived from host `gh auth token` — no extra config needed.	host `gh` token
`CW_SSH_PATH`	Host `~/.ssh` (mounted ro)	`${HOME}/.ssh`
`CW_CONTAINER_DATA_HOME`	`COPILOT_DATA_HOME` inside the container	`/data/copilot`
`CW_CONFIG_FLAG`	`--config <abs path>` if `.gru/config.yml` exists	`--config /tools/gru/.gru/config.yml`

Override any of these by exporting before source ./gru.

The docker contract

Every container run goes through one helper:

_run_cw_docker <interactive:true|false> <command-string> [host_workspace_dir]
_cw_dock      <command-string> [host_workspace_dir]   # interactive (-it)
_cw_dock_cmd  <command-string> [host_workspace_dir]   # non-interactive

Standard mounts on every run:

Mount / var	Path in container	Mode
`PROJECT_TOP`	`/tools/gru`	ro (tooling)
`PROJECT_TOP/data`	`/tools/gru/data`	rw overlay
`PROJECT_TOP/docs`	`/tools/gru/docs`	rw overlay
`host_workspace_dir` (if given)	`/workspace`	rw (your code)
`~/.ssh`	`/root/.ssh`	ro
`~/.azure`	`/root/.azure`	ro (auto, if present)
`~/.gitconfig`	`/root/.gitconfig`	ro (auto, if present)
`gru-data`	`/data/copilot` (`COPILOT_DATA_HOME`)	volume
`gru-logs`	`/logs`	volume
`GH_TOKEN`, `GH_HOST`	env	—

Plugin conventions:

Use _cw_quote "$@" to safely embed user args in the command string.
Prefix with ${CW_AUTH_BOOTSTRAP} so gh/copilot are authenticated before your command runs.
Use _cw_dock for interactive (TTY), _cw_dock_cmd otherwise.

Commands

`copilot` — Copilot CLI over host code

copilot shell [--dir PATH]                    # interactive Copilot CLI
copilot run [--dir PATH] "prompt" [args...]   # one non-interactive prompt

`gh-watch` — autopilot over a GitHub project board

gh-watch [DIR] {run|start|stop|status} [BOARD] [--dir PATH] [--port N] [--dry-run] [extra args...]

DIR (optional) is the name of a config subdirectory inside the workspace containing a config.yml. It differentiates container names so multiple boards can run simultaneously:

gh-watch hil-stress start           # daemon + dashboard at http://localhost:9300
gh-watch hil-stress status          # running state, recent logs, dashboard URL
gh-watch hil-stress stop            # stop daemon and dashboard
gh-watch hil-stress run --dry-run   # foreground, no sessions started

run executes in the foreground. start launches a background daemon container and opens a live dashboard in the browser:

Pipeline columns — all issues grouped by stage, polled from GitHub every 30 s
Active badge — current issue # and stage with a spinner
Completed this run — ✓/✗ chips per finished issue
Live log — SSE-streamed docker logs, colour-coded

Idle-poll interval

The daemon re-queries the board immediately after finishing each issue. When no actionable issue is found it sleeps for poll_interval seconds, then re-queries.

Setting	Behaviour
`poll_interval: N` (config)	Sleep N seconds when board is idle (default: 300)
`--poll-interval N` (CLI, takes precedence)	Same, overrides config
`poll_interval: 0` / `--poll-interval 0`	Single-pass — exits when board empties

# .gru/config.yml
watcher:
  poll_interval: 300   # 0 = single-pass, default = 300

gh-watch hil-stress run --poll-interval 0   # run once and exit (CI)
gh-watch hil-stress start --poll-interval 60  # daemon, poll every 60 s

Stage prompts for `gh-watch`

Each board column maps to a prompt file. watcher-run.sh uses two-level resolution:

Consumer override — ${watcher.prompts_dir}/${stage}.md (relative to /workspace)
Built-in fallback — stage-prompts/${stage}.md in the docker-gru-env tooling

Prompt files are envsubst templates. Available variables: ${ISSUE_NUM}, ${REPO}, ${GH_HOST}, ${ISSUE_STAGE}.

Parent/epic issues (sub-issue guard)

Issues with sub-issues are never processed — they are human-managed trackers. The guard applies the human-only label automatically and skips the issue.

`md-watch` — autopilot from an Obsidian Kanban markdown board

md-watch {run|start|stop|status} <board.md> [--dir PATH] [--column NAME] [--port N] [--dry-run] [--apply] [-- copilot-args...]

Reads an Obsidian Kanban board (.md file) and starts one Copilot session per open card (- [ ]) in the actionable column (default Todo). --apply marks processed cards done.

md-watch start my-board.md              # daemon + dashboard at http://localhost:9301
md-watch status my-board.md
md-watch stop my-board.md
md-watch run my-board.md --dry-run      # foreground

`cost` — reporting & attribution

cost report [--format html] [--output PATH] [flags]   # python3 src/cost-report.py
cost link [--apply] [flags]                            # python3 src/cost-link.py
cost dashboard [--regen-only|--publish-only] [flags]   # scripts/build-dashboard.sh

`data` — mirror JSONL → attributions DB

data update    # cost-link --apply: upsert attributions into the volume DB
data preview   # dry-run, no DB writes

Quick start (standalone)

git clone https://<your-ghe-host>/<owner>/copilot-workflow ~/tools/gru
cd ~/tools/gru

# Register the sessionEnd hook
./scripts/install-hook.sh

# Install skills
mkdir -p ~/.copilot/skills/{issue-start,session-handoff,cost-report}
cp skills/issue-start/SKILL.md     ~/.copilot/skills/issue-start/
cp skills/session-handoff/SKILL.md ~/.copilot/skills/session-handoff/
cp skills/cost-report/SKILL.md     ~/.copilot/skills/cost-report/

# Backfill historical sessions
python3 src/cost-retrospective.py

# Generate dashboard
python3 src/cost-report.py --format html
open docs/cost-dashboard.html

`COPILOT_DATA_HOME`

All src/*.py scripts honour this env var. When set, JSONL logs and session-state are read from and written to that directory instead of ~/.copilot/.

COPILOT_DATA_HOME=/tmp/test-copilot python3 src/cost-report.py --text
COPILOT_DATA_HOME=$PWD/data python3 src/cost-report.py --format html

Consumer repo setup

Create .gru/config.yml in your consumer repo:

gh_host: your-ghe-host.example.com   # GitHub Enterprise hostname (or github.com)
data_repo: owner/your-repo
pages_repo: owner/pages-repo

project:
  owner: your-org-or-user
  number: 1

watcher:
  prompts_dir: stage-prompts   # consumer stage handlers (relative to this file)
  poll_interval: 300           # seconds to sleep when board is idle (0 = single-pass)
  max_issues: 50

Add or override stage handlers:

mkdir -p .gru/stage-prompts
cp stage-prompts/Todo.md .gru/stage-prompts/Todo.md
# create .gru/stage-prompts/HW-Test.md for hardware testing stages

Run ./scripts/install-hook.sh once to register the sessionEnd hook.

Source the environment (if using as a submodule):

git submodule add <copilot-workflow-url> copilot-workflow
ln -s copilot-workflow/gru gru
source ./gru

Using as a submodule

PROJECT_TOP resolves to the submodule checkout (not the parent repo), so the image build context and all mounts always point at copilot-workflow.

consumer-repo/                    HOST
  copilot-workflow/  ← submodule  mounted ro at /tools/gru
  .gru/
    config.yml
    stage-prompts/  ← overrides   accessible at /workspace/.gru/stage-prompts
  (code to edit)                  mounted rw at /workspace

Extension model

copilot-workflow is a base class. Consumer projects override stage prompts and extend Copilot instructions. The pattern is C++ virtual methods: base provides defaults, consumer overrides what it needs.

Three-level hierarchy

copilot-workflow/              ← base class
  stage-prompts/Todo.md         generic workflow (read issue, implement, review, handoff)
  stage-prompts/In Progress.md  generic resume workflow

~/ws/platform/                 ← workspace subclass (consumer)
  .gru/config.yml   wires watcher-run to platform prompts
  stage-prompts/Todo.md         overrides base: adds multi-repo paths, context file
                                 locations, branch rules, GHE host
  .github/copilot-instructions.md  platform-level Copilot rules

roomboard-linux/               ← repo subclass (most specific)
  .github/copilot-instructions.md  Yocto/BitBake rules, meta layer layout, branch policy

Stage prompts vs. Copilot instructions

	Stage prompt (`Todo.md`)	Copilot instructions (`.github/…`)
Loaded by	`watcher-run.sh` before session starts	Copilot CLI from cwd git root
Scope	One issue session	All sessions in that repo
Override mechanism	`watcher.prompts_dir` in `.gru/config.yml`	One file per git repo
Best for	Workflow steps, context injection	Repo conventions, file layout, safety rules

Writing a consumer stage prompt

A consumer Todo.md embeds the base workflow as a "super" call:

<!-- Consumer preamble -->
You are working in the `~/ws/platform/` multi-repo workspace.
Branch base: kirkstone-dev  (never target kirkstone directly)
GH host:     github.com

<!-- Base workflow (super()) -->
You are working on issue #${ISSUE_NUM} in repo ${REPO} (GH_HOST=${GH_HOST}).
... (rest of base Todo.md embedded here)

To add a stage that doesn't exist in the base (e.g. HW-Test), simply create .gru/stage-prompts/HW-Test.md — no base change needed.

Watcher runs (direct, without `gru`)

./scripts/watcher-run.sh --config .gru/config.yml
./scripts/watcher-run.sh --repo owner/repo --host your-ghe.example.com --project 8

Flags:

Flag	Description
`--config PATH`	Load repo/project/watcher settings from YAML
`--dry-run`	Print sessions that would run, do nothing
`--log-dir DIR`	Write per-issue + full-run logs to DIR
`--poll-interval N`	Sleep N seconds when board is idle (0 = single-pass)
`--max N`	Safety cap on total issues processed
`--max-per-issue N`	Max retry attempts per issue (prevents retry storms)
`--resume`	Skip already-completed issues from a previous run
`--working-dir PATH`	Run sessions from PATH (multi-repo workspace support)

Service container (CI / unattended)

For CI runners or cron jobs without a host environment, use docker/entrypoint.sh. It clones the consumer workspace and runs watcher-run.sh fully unattended.

Required env vars:

Variable	Description
`GH_TOKEN`	Personal access token with `repo` + `project` scopes
`WORKSPACE_REPO`	`owner/repo` slug of the consumer workspace
`GH_HOST`	GitHub hostname (default: `github.com`)
`LINKED_REPOS`	Space-separated `name=owner/repo` pairs cloned into `/workspace/`
`OVERNIGHT_ARGS`	Extra flags passed verbatim to `watcher-run.sh`

docker-compose.yml (consumer project):

services:
  watcher:
    image: gru:local
    build:
      context: /path/to/copilot-workflow
      dockerfile: docker/Dockerfile
    environment:
      GH_TOKEN: ${GH_TOKEN}
      GH_HOST: github.com
      WORKSPACE_REPO: owner/platform
      LINKED_REPOS: roomboard-linux=owner/roomboard-linux
      OVERNIGHT_ARGS: --project 8
    volumes:
      - gru-data:/data/copilot
      - gru-logs:/logs

volumes:
  gru-data:
  gru-logs:

Cost dashboard

GH_HOST=your.ghe.com ./scripts/build-dashboard.sh

Steps: regenerate HTML dashboards → sync Cost ($) field on project boards → push to Pages.

Flag	Description
`--regen-only`	Regenerate dashboards only, skip publish
`--publish-only`	Push existing `docs/` without regenerating
`--dry-run`	Print what would happen, write nothing

Config fields

gh_host: github.com
data_repo: owner/copilot-workflow
pages_repo: owner/owner.github.com

pages:
  branch: main

project:
  owner: my-org
  number: 8

# Normalise renamed/moved repos to canonical names used on project boards
repo_aliases:
  "old-owner/old-repo": "new-owner/new-repo"

# Fallback project for sessions whose issue refs don't match any board issue
repo_projects:
  "owner/platform-workspace": 13
  "owner/daily-dev-tracker": 8

Cost attribution pipeline

Step 1 — Auto-attribution

GH_HOST=github.com python3 src/cost-link.py          # preview
GH_HOST=github.com python3 src/cost-link.py --apply  # write

Strategies (in order): existing issue_refs → commit SHA → branch name → issue number in session title.

Step 2 — Manual attribution

Edit .gru/manual-attributions.yml:

1899dfe2: 25          # known issue number
298fe3ff:             # known project, unknown issue
  issue: -1
  project: 3
64afe9c8:             # leave blank to keep in Unlinked

python3 src/cost-link-manual.py --apply

Step 3 — Find new unlinked sessions

./scripts/identify-unlinked.sh          # auto-attribute + append stubs to YAML
./scripts/identify-unlinked.sh --apply  # open editor → preview → apply → rebuild

Flag	Description
`--no-auto`	Skip auto-attribution
`--dry-run`	Print stubs without writing
`--min-cost N`	Only surface sessions ≥ N USD (default: 0.01)

Step 4 — Sync costs to project boards

Writes per-issue cost totals to a Cost ($) field on GitHub Projects V2 (created automatically).

GH_HOST=github.com python3 src/cost-board-sync.py --project-owner my-org --all-projects

Also runs automatically inside build-dashboard.sh.

Full pipeline

GH_HOST=github.com ./scripts/identify-unlinked.sh
python3 src/cost-link-manual.py --apply
GH_HOST=github.com ./scripts/build-dashboard.sh

Data & DB model

The sessionEnd hook (docker/hooks.json → src/cost-sync.py) writes cost JSONL to COPILOT_DATA_HOME=/data/copilot (the data volume).
The attributions DB lives on the volume (/data/copilot/attributions.db) — writable because the repo mount is ro.
_seed_cw_data seeds the volume from committed data/ on first run; never overwrites live data.

Records in ~/.copilot/cost-log.jsonl:

{
  "schema_version": 1,
  "session_id": "abc123",
  "confidence": "exact",
  "repository": "owner/repo",
  "branch": "feat/my-feature",
  "started_at": "2026-06-01T10:00:00Z",
  "ended_at": "2026-06-01T11:30:00Z",
  "issue_refs": [{"issue": 42, "confidence": "exact"}],
  "model_metrics": {"claude-sonnet-4.6": {"input_tokens": 500000, "output_tokens": 8000}},
  "est_cost_usd": 1.85,
  "total_premium_requests": 12
}

Scripts reference

Script	Description
`scripts/build-dashboard.sh`	Regen all dashboards, sync board costs, publish to Pages
`scripts/identify-unlinked.sh`	Auto-attribute + find new unlinked sessions
`scripts/watcher-run.sh`	Multi-issue autopilot loop
`scripts/publish-ghpages.sh`	Push `docs/` to Pages branch only
`scripts/install-hook.sh`	Register `sessionEnd` hook
`scripts/update-data.sh`	Sync JSONL into `data/` and commit (`--push` to push)

Tool	Description
`src/cost-sync.py`	`sessionEnd` hook — appends one record to JSONL
`src/cost-retrospective.py`	Backfill historical sessions to JSONL
`src/cost-report.py`	Generate HTML dashboards and text reports
`src/cost-link.py`	Auto-attribute sessions via commits/PRs/branches
`src/cost-link-manual.py`	Apply hand-written attributions from YAML
`src/cost-identify-unlinked.py`	Find unlinked sessions, append stubs to YAML
`src/cost-board-sync.py`	Sync per-issue costs to GitHub Projects V2
`src/attributions_db.py`	SQLite attribution store (`attributions.db`)
`src/pricing.py`	USD cost engine

Extending: add a new plugin

Create plugins/<name>.sh.
Register and define:

hello_init() {
    register_plugin_command "hello" "hello" \
        "Example plugin" \
        "hello [name] - print a greeting from inside the container"
}

hello() {
    local extra
    extra=$(_cw_quote "$@")
    _cw_dock_cmd "${CW_AUTH_BOOTSTRAP}; cd /tools/gru && echo hello${extra}"
}

source ./gru — load_plugins auto-discovers it.

The image (`docker/Dockerfile`)

ubuntu:24.04 + python3/pip (PyYAML), git, curl, jq, gettext-base, openssh-client, nodejs/npm, gh CLI.
Installs the standalone GitHub Copilot CLI via npm install -g @github/copilot.
Installs a gh-copilot shim so gh copilot -- … (used by watcher-run.sh) keeps working.
ENV COPILOT_DATA_HOME=/data/copilot; declares volumes for /data/copilot, /logs, /workspace.

Gotchas & limitations

Rebuild after image changes. Run source ./gru --rebuild or docker rmi gru:local.
macOS Docker Desktop file sharing. Workspace dirs must be under $HOME. /var/folders temp dirs produce an empty /workspace.
copilot is a shell function after sourcing. It shadows any host copilot binary. Open a fresh shell or unset -f copilot to use the host CLI.
zsh. Supported (${(%):-%x} fallback). Uses bashisms (local -a, [[ =~ ]]) that zsh also accepts.
CW_EXTRA_DOCKER_FLAGS — word-split on IFS; paths with spaces break it. Use auto-mounts for ~/.azure/~/.gitconfig instead.

Pricing

Pricing data is read from (in order):

COPILOT_COST_PRICING env var (path to YAML)
~/.copilot/cost-cache/pricing.yaml
~/tools/copilot-cost/pricing.snapshot.yaml
src/pricing.snapshot.yaml (bundled fallback)

Refresh: npm run refresh-pricing in ~/tools/copilot-cost/ (requires devartifex/copilot-cost).

Name		Name	Last commit message	Last commit date
.gru		.gru
core		core
docker		docker
lib		lib
plugins		plugins
scripts		scripts
skills		skills
src		src
stage-prompts		stage-prompts
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
README.md		README.md
gru		gru
install-skills.sh		install-skills.sh
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

copilot-workflow

What it does

Architecture

The gru environment

File layout

Boot sequence

Key configuration (core/config.sh)

The docker contract

Commands

copilot — Copilot CLI over host code

gh-watch — autopilot over a GitHub project board

Idle-poll interval

Stage prompts for gh-watch

Parent/epic issues (sub-issue guard)

md-watch — autopilot from an Obsidian Kanban markdown board

cost — reporting & attribution

data — mirror JSONL → attributions DB