Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
root = true

[*]
charset = utf-8
end_of_line = lf
indent_style = space
indent_size = 2
insert_final_newline = true
trim_trailing_whitespace = true

[*.md]
trim_trailing_whitespace = false
1 change: 1 addition & 0 deletions .envrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
use flake
92 changes: 92 additions & 0 deletions .github/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# `.github/` — workflows for this template

Reusable GitHub Actions workflows callable from any dryvist Cribl pack via
`uses: dryvist/cc-edge-pack-template/.github/workflows/<file>@main`. Caller
workflows in this repo (`test.yml`, `release.yml`, `release-please.yml`) wire
them up to GitHub events.

## Installation

These workflows install themselves implicitly when a new pack scaffolds from
`dryvist/cc-edge-pack-template` (`gh repo create --template`). The caller
workflows under `.github/workflows/` are copied verbatim; the reusable
workflows they reference resolve at runtime against
`dryvist/cc-edge-pack-template@main`.

If you're adding a workflow to a non-template repo manually, copy the relevant
caller from this directory into your repo's `.github/workflows/` and adjust
the `uses:` ref if you want to pin a specific template version (default
`@main` follows the template's main branch).

## Usage

Once installed, the workflows trigger automatically:

- Open a PR touching `default/`, `data/samples/`, `tests/`, or `package.json`
→ `test.yml` runs validate + Vitest.
- Push to `main` → `release-please.yml` opens or updates the release PR.
- Merge a release PR → release-please tags the version, which triggers
`release.yml` → builds the `.crbl` and publishes to GitHub Releases.

No manual invocation is needed. Override defaults by editing the caller
workflow's `with:` block (e.g., bump `cribl_version` to test a specific
upstream Cribl release).

## Reusable workflows

### `cribl-pack-test.yml`

Validates pack structure + runs the TypeScript Vitest suite against a
`cribl/cribl` service container.

| Input | Required | Default | Purpose |
|---|---|---|---|
| `pack_type` | yes | — | `edge` or `stream` (drives validator naming convention + required-fields assertion) |
| `cribl_version` | no | `latest` | `cribl/cribl` Docker tag |
| `node_version` | no | `lts/*` | Node.js version |
| `yq_version` | no | `4.44.5` | Pinned `mikefarah/yq` version |

Jobs:

1. **validate** — installs `yq` (pinned via `dcarbone/install-yq-action`),
lints YAML (`frenck/action-yamllint`, config from `.yamllint.yml`), runs
`scripts/validate-pack-structure.sh`.
2. **test** — sets up Node + pnpm (cached), installs deps, runs Biome lint,
typechecks, waits for Cribl health endpoint
(`iFaxity/wait-on-action`), runs `pnpm run test`.

### `cribl-pack-release.yml`

Builds a `.crbl` tarball via `scripts/build-crbl.sh` and publishes it to
GitHub Releases (`softprops/action-gh-release`).

| Input | Required | Default | Purpose |
|---|---|---|---|
| `additional_files` | no | `''` | Space-separated extra files to include in tarball; `LICENSE` auto-included if present |

Triggered by tag push in the calling workflow (typically by release-please
when a release PR merges).

## Caller workflows in this repo

| File | Trigger | Calls |
|---|---|---|
| `workflows/test.yml` | PR + push to main (paths-filtered) | `cribl-pack-test.yml` (this repo) |
| `workflows/release.yml` | tag push matching `v*` | `cribl-pack-release.yml` (this repo) |
| `workflows/release-please.yml` | push to main | `_release-please.yml` (inherited from `JacobPEvans/.github`) |

## Updating reusable workflow inputs

Adding/renaming an input is a breaking change for every consumer pack. Use the
`workflow_call.inputs.<name>.default` field to keep the change backward
compatible whenever possible. Bump major version of the workflow only if you
absolutely cannot.

## Pinning external actions

Per [`SECURITY.md`](https://github.com/dryvist/.github/blob/main/SECURITY.md):

- Trusted (`actions/*`, `pnpm/action-setup`, `softprops/action-gh-release`):
semver tag pins (`@v4`, etc.)
- Untrusted (everything else): SHA pins. Renovate (per the org's
`renovate.json`) auto-converts version tags to SHA on first run.
25 changes: 25 additions & 0 deletions .github/workflows/release-please.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: release-please

# Inherits the canonical release-please pipeline from JacobPEvans/.github,
# which enforces the org-wide major-bump block and uses the GitHub App token
# so release-please PRs trigger downstream workflows. See
# dryvist/.github/CLAUDE.md for the inheritance chain.

on:
push:
branches: [main]

permissions:
contents: write
pull-requests: write

jobs:
release-please:
if: github.event.repository.is_template == false
uses: JacobPEvans/.github/.github/workflows/_release-please.yml@main
# The inherited workflow's input is named GH_ACTION_JACOBPEVANS_APP_ID for
# historical reasons. dryvist exposes a generic GH_APP_ID org secret and
# forwards it here at the boundary — repo readers only see the generic name.
secrets:
GH_ACTION_JACOBPEVANS_APP_ID: ${{ secrets.GH_APP_ID }}
GH_APP_PRIVATE_KEY: ${{ secrets.GH_APP_PRIVATE_KEY }}
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ permissions:
jobs:
release:
if: github.event.repository.is_template == false
uses: dryvist/.github/.github/workflows/cribl-pack-release.yml@main
uses: dryvist/cc-edge-pack-template/.github/workflows/cribl-pack-release.yml@main
7 changes: 4 additions & 3 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ permissions:

jobs:
test:
# Skip on the template repo itself; consumer repos have is_template: false.
if: github.event.repository.is_template == false
uses: dryvist/.github/.github/workflows/cribl-pack-test.yml@main
# Runs on the template too — the template now ships a working demo
# passthrough pipeline + fixture so the harness gets exercised here
# before consumer repos inherit it.
uses: dryvist/cc-edge-pack-template/.github/workflows/cribl-pack-test.yml@main
with:
# CHANGE per pack: 'edge' for Cribl Edge packs, 'stream' for Cribl Stream packs.
pack_type: edge
16 changes: 10 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
# macOS
.DS_Store
*.swp
*.bak

# Build artifacts
*.crbl

# Python
.venv/
**/.venv/
__pycache__/
*.pyc
.pytest_cache/
# Node / TypeScript
node_modules/
**/node_modules/
dist/
*.tsbuildinfo

# Nix / direnv
.direnv/
.envrc.cache

# Editor / OS
.idea/
Expand Down
1 change: 1 addition & 0 deletions .release-please-manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{ ".": "0.0.1" }
10 changes: 10 additions & 0 deletions .yamllint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
extends: default
rules:
line-length: disable
comments-indentation: disable
comments: disable
truthy:
level: warning
indentation:
level: warning
document-start: disable
103 changes: 18 additions & 85 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,93 +1,26 @@
# CLAUDE.md — guidance for AI assistants working in this repo
## What this repo is

This file is read by Claude Code (and other AI assistants supporting `CLAUDE.md`) on every session. It encodes the guardrails for working in this template — and, by inheritance, in any pack scaffolded from it.
Cribl Edge pack scaffolded from
[`dryvist/cc-edge-pack-template`](https://github.com/dryvist/cc-edge-pack-template).
Eventual purpose: collect Claude Code telemetry (session JSONL transcripts +
OpenTelemetry metrics) and forward to a Cribl Stream worker group.

## Repository Type
## Current state

This is a **template repository** for new Cribl Edge / Stream packs. It is consumed via `gh repo create --template`. The files here become the starting point for every downstream pack.
Ships only the template's demo passthrough pipeline + fixture so the
inherited Vitest harness has something to run end-to-end. Real
`claude-code-otel` and `claude-code-session-logs` pipelines (with eval
functions, real fixtures, captured samples) land in subsequent PRs.

## Generic vs Pack-Specific
## Sources of truth

The single most important rule: **distinguish generic files from pack-specific files.**

**Generic** (DO NOT modify in pack repos — only in this template):
- `tests/cribl_client.py`
- `tests/conftest.py`
- `tests/test_pipelines.py`
- `tests/test_routes.py`
- `tests/requirements.txt`
- `Makefile`
- `docker-compose.yml`
- `.github/workflows/test.yml` (only the `pack_type:` value changes per pack)
- `.github/workflows/release.yml`

If you find yourself wanting to modify any of the above in a pack repo, **stop**. Either:

1. The change belongs in this template (open a PR here, then propagate to packs), or
2. The change should be expressed as fixture data, not code.

**Pack-specific** (free to modify per-pack):
- `package.json` (name, version, displayName, tags, description)
- `default/pack.yml` (logo)
- `default/inputs.yml` (sources)
- `default/pipelines/route.yml` (routes)
- `default/pipelines/<name>/conf.yml` (pipeline functions)
- `default/samples.yml` (sample catalog)
- `data/samples/*.json` (sample events)
- `tests/fixtures/<pipeline>/*.json` (test fixtures)
- `README.md` (describe your specific pack)
- `LICENSE` (use Apache-2.0 unless instructed otherwise)

## Validator Rules (vct-cribl-pack-validator)

Always enforce these — they are non-negotiable per the [validator skill](https://github.com/VisiCore/vct-cribl-pack-validator):

| Rule | What it means |
| Layer | Where |
|---|---|
| Pack ID format | `cc-edge-<source>-io` for Edge, `cc-stream-<source>-io` for Stream |
| No pipeline named `main` | All pipelines must have descriptive names |
| All routes use `output: __group` | Never `input_id` (breaks on source rename) |
| All sources have `metadata.datatype` | So route filters can match |
| Filters must be dynamic | Never literal `false` / `0` |
| No hardcoded paths | Use environment variables (`$MY_LOG_PATH`) |
| No hardcoded credentials | Use Cribl secrets |
| PII fields masked | `email`, `username`, `*_id`, `src_ip`, etc. before destinations |

## Fixture Convention

When adding tests, follow filesystem convention — no Python edits required:

```
tests/fixtures/<pipeline-name>/<case>.json # input
tests/fixtures/<pipeline-name>/<case>.expected.json # optional expected output (partial match)
```

The generic `test_pipelines.py` auto-discovers and parametrizes one test per `<case>.json` it finds. If `<case>.expected.json` is missing, the case is a smoke test (asserts non-empty output only). When you add an expected file, the assertions tighten automatically.

Prefer richer fixtures over richer Python. If the assertion can't be expressed as a partial-match expected event, it probably shouldn't be a test — consider whether it's really a pipeline behavior or something else.

## Don't Invent — Reuse

Per the user's rules:

- **Use existing Cribl tooling** — `cribl pipe`, the management API, official Docker images. Don't reinvent.
- **Use existing third-party Actions** — `softprops/action-gh-release`, `rlespinasse/github-slug-action`, `actions/setup-python`. Don't write custom packaging shell scripts.
- **Use the criblpacks pattern** — that's where `cribl_client.py` came from. When extending, mirror their idioms.
- **Use vct-cribl-pack-validator** — for deep structural validation, not custom-rolled YAML parsers.

## Workflow

For any pack work:

1. `/refresh-repo` then create a worktree for the change (per user's global CLAUDE.md).
2. Modify only pack-specific files (see lists above).
3. `make test` locally before committing.
4. `make validate` before tagging a release.
5. Tag `vX.Y.Z` to trigger the release workflow.
| Org-wide policy (TS-everywhere, Biome, Vitest, secrets, releases) | [`dryvist/.github`](https://github.com/dryvist/.github) |
| Test harness mechanics, file boundary, validator rules | [`dryvist/cc-edge-pack-template/docs/`](https://github.com/dryvist/cc-edge-pack-template/tree/main/docs) |

## When in Doubt
## Top-level rules

- Read [`VisiCore/cc-edge-claude-code-io`](https://github.com/VisiCore/cc-edge-claude-code-io) — the gold-standard reference pack.
- Read [`criblpacks/cribl-palo-alto-networks`](https://github.com/criblpacks/cribl-palo-alto-networks) — Cribl's own test pattern reference.
- Read [`VisiCore/vct-cribl-pack-validator`](https://github.com/VisiCore/vct-cribl-pack-validator) — the authoritative ruleset.
- Don't add scripts. If you're tempted to write a script, ask first whether a Cribl-native or GitHub Action equivalent already exists.
- Don't modify generic files here — open a PR against the template repo and let it propagate.
- Don't tag versions; release-please proposes them via PR.
- Don't write inline scripts in workflows — extract to `scripts/*.sh` or use a community action.
30 changes: 17 additions & 13 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,15 @@ SHELL := /bin/bash
PACK_NAME := $(shell jq -r '.name // "unknown-pack"' package.json 2>/dev/null)
PACK_VERSION := $(shell jq -r '.version // "0.0.0"' package.json 2>/dev/null)
CRBL_FILE := $(PACK_NAME)-$(PACK_VERSION).crbl
VENV := .venv

.PHONY: help install build docker-up docker-down test validate clean
.PHONY: help install build docker-up docker-down test typecheck lint format validate clean

help: ## Show this help
@grep -hE '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | \
awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-15s\033[0m %s\n", $$1, $$2}'

$(VENV):
python3 -m venv $(VENV)
$(VENV)/bin/pip install --upgrade pip
$(VENV)/bin/pip install -r tests/requirements.txt

install: $(VENV) ## Create venv and install Python test dependencies
install: ## Install Node test dependencies (pnpm)
cd tests && pnpm install

build: ## Build .crbl artifact (mirrors what release.yml does in CI)
@INCLUDE="data default package.json README.md"; \
Expand All @@ -25,7 +20,7 @@ build: ## Build .crbl artifact (mirrors what release.yml does in CI)
@echo "Built: $(CRBL_FILE)"
@du -h "$(CRBL_FILE)"

docker-up: ## Start cribl/cribl Docker container (Stream mode supports both Edge and Stream packs)
docker-up: ## Start cribl/cribl Docker container (Stream supports both Edge and Stream packs)
docker compose up -d
@echo "Waiting for Cribl to be ready..."
@for i in $$(seq 1 30); do \
Expand All @@ -41,8 +36,17 @@ docker-up: ## Start cribl/cribl Docker container (Stream mode supports both Edge
docker-down: ## Stop and remove the Docker container
docker compose down -v

test: install ## Run pytest test suite (requires Docker; run 'make docker-up' first)
$(VENV)/bin/python -m pytest tests/ -v
test: ## Run Vitest test suite (requires Docker; run 'make docker-up' first)
cd tests && pnpm run test

typecheck: ## Type-check TypeScript
cd tests && pnpm run typecheck

lint: ## Lint with Biome (lint + format check)
cd tests && pnpm run lint

format: ## Auto-format with Biome
cd tests && pnpm run format

validate: build ## Build pack and instruct on validator usage
@echo ""
Expand All @@ -51,7 +55,7 @@ validate: build ## Build pack and instruct on validator usage
@echo "To run vct-cribl-pack-validator (27+ structural checks), from the validator repo:"
@echo " cd ~/git/vct-cribl-pack-validator/main && claude /validate-pack $$PWD/$(CRBL_FILE)"

clean: ## Remove build artifacts, venv, and stop Docker
clean: ## Remove build artifacts, node_modules, and stop Docker
rm -f *.crbl
rm -rf $(VENV) tests/__pycache__ tests/.pytest_cache .pytest_cache
rm -rf tests/node_modules tests/dist
-docker compose down -v 2>/dev/null
Loading
Loading