Orq Evaluate
ActionsTags
(2)Run evaluatorq evaluation files in your CI/CD pipeline.
Source Repository: This action is maintained in the orqkit monorepo. The source code for the CLI and evaluatorq framework lives there.
This action requires you to install your project dependencies before running evaluations.
- name: Run evaluations
uses: orq-ai/cli-evaluate@v1
with:
pattern: "**/*.eval.ts"
orq-api-key: ${{ secrets.ORQ_API_KEY }}- name: Run evaluations
uses: orq-ai/cli-evaluate@v1
with:
pattern: "**/*.eval.py"
orq-api-key: ${{ secrets.ORQ_API_KEY }}| Input | Description | Required | Default |
|---|---|---|---|
pattern |
Glob pattern for evaluation files (e.g., **/*.eval.ts or **/*.eval.py) |
Yes | - |
orq-api-key |
Orq API key for authentication and tracing | No | - |
orq-base-url |
Orq API base URL | No | https://my.orq.ai |
working-directory |
Working directory to run evaluations from | No | . |
disable-tracing |
Disable OTEL tracing even when API key is set | No | false |
otel-endpoint |
Custom OTEL endpoint for traces | No | - |
| Output | Description |
|---|---|
passed |
Whether all evaluations passed (true or false) |
name: Evaluations
on:
push:
branches: [main]
pull_request:
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run evaluations
uses: orq-ai/cli-evaluate@v1
with:
pattern: "evaluations/**/*.eval.ts"
orq-api-key: ${{ secrets.ORQ_API_KEY }}When orq-api-key is provided, traces are automatically sent to the Orq platform. You can:
- Use
disable-tracing: trueto disable tracing - Use
otel-endpointto send traces to a custom OTEL collector
MIT - See LICENSE
- orqkit - Source repository with evaluatorq framework and CLI
- @orq-ai/evaluatorq - TypeScript evaluation framework
- evaluatorq-py - Python evaluation framework
Orq Evaluate is not certified by GitHub. It is provided by a third-party and is governed by separate terms of service, privacy policy, and support documentation.
Tags
(2)Orq Evaluate is not certified by GitHub. It is provided by a third-party and is governed by separate terms of service, privacy policy, and support documentation.