Skip to content

README.md

Latest commit

 

History

History
88 lines (63 loc) · 2.32 KB

README.md

File metadata and controls

88 lines (63 loc) · 2.32 KB

Orq Evaluate GitHub Action

Run evaluatorq evaluation files in your CI/CD pipeline.

Source Repository: This action is maintained in the orqkit monorepo. The source code for the CLI and evaluatorq framework lives there.

Prerequisites

This action requires you to install your project dependencies before running evaluations.

Usage

TypeScript

- name: Run evaluations
  uses: orq-ai/cli-evaluate@v1
  with:
    pattern: "**/*.eval.ts"
    orq-api-key: ${{ secrets.ORQ_API_KEY }}

Python

- name: Run evaluations
  uses: orq-ai/cli-evaluate@v1
  with:
    pattern: "**/*.eval.py"
    orq-api-key: ${{ secrets.ORQ_API_KEY }}

Inputs

Input Description Required Default
pattern Glob pattern for evaluation files (e.g., **/*.eval.ts or **/*.eval.py) Yes -
orq-api-key Orq API key for authentication and tracing No -
orq-base-url Orq API base URL No https://my.orq.ai
working-directory Working directory to run evaluations from No .
disable-tracing Disable OTEL tracing even when API key is set No false
otel-endpoint Custom OTEL endpoint for traces No -

Outputs

Output Description
passed Whether all evaluations passed (true or false)

Example Workflow

name: Evaluations

on:
  push:
    branches: [main]
  pull_request:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run evaluations
        uses: orq-ai/cli-evaluate@v1
        with:
          pattern: "evaluations/**/*.eval.ts"
          orq-api-key: ${{ secrets.ORQ_API_KEY }}

Tracing

When orq-api-key is provided, traces are automatically sent to the Orq platform. You can:

  • Use disable-tracing: true to disable tracing
  • Use otel-endpoint to send traces to a custom OTEL collector

License

MIT - See LICENSE

Related