ci: add beval behavioral evaluation workflow for dt-coach agent

[eedorenko]

Description

Adds a behavioral evaluation (beval) CI workflow for the dt-coach agent using GitHub Copilot CLI over ACP (TCP). The workflow:

• Starts two Copilot CLI instances (agent on port 3000, judge on port 3001)
• Runs beval evaluations against cases defined in beval/cases/
• Uploads results as a workflow artifact

Also pins all GitHub Actions dependencies to SHA hashes for supply chain security, and installs beval from the default branch of the vyta/beval repo.

Sample eval run: https://github.com/eedorenko/hve-core/actions/runs/23311489579/job/67799722616

======================================================================
  SCORECARD
======================================================================
  Overall: 0.81  (30/30 cases passed)

  Metric                Score  Bar
  -------------------- ------  ------------
  latency               0.86  [#########-]
  quality               0.78  [########--]

  Case                                      Score  Status
  ---------------------------------------- ------  ------
  Response follows Think/Speak/Empower ...  0.66  + PASS
  Keep responses concise — no methodolo...  0.86  + PASS
  End with choices not directives           0.91  + PASS
  Work WITH users, not FOR them             0.95  + PASS
  Do not prescribe specific solutions t...  0.93  + PASS
  Stay curious and supportive when user...  0.81  + PASS
  Method 1: Assess whether request is f...  0.93  + PASS
  Method 1: Guide stakeholder identific...  0.82  + PASS
  Method 2: Help plan systematic research   0.68  + PASS
  Method 3: Guide pattern recognition f...  0.81  + PASS
  Method 4: Facilitate divergent ideation   0.65  + PASS
  Method 5: Guide concept creation for ...  0.65  + PASS
  Method 6: Encourage scrappy constrain...  0.86  + PASS
  Method 7: Guide technical feasibility...  0.65  + PASS
  Method 8: Structure user testing for ...  0.65  + PASS
  Method 9: Guide continuous optimizati...  0.63  + PASS
  Start with broad hints when user is s...  0.73  + PASS
  Escalate hints when user remains stuck    0.90  + PASS
  Accept backward transitions between m...  0.83  + PASS
  Announce method shifts transparently      0.84  + PASS
  Avoid multiple-choice question lists      0.81  + PASS
  Do not change method focus without an...  0.95  + PASS
  Resume session with state context         0.63  + PASS
  Ask for project slug during initializ...  0.94  + PASS
  Gather role, team, and method focus d...  0.92  + PASS
  Default to Method 1 for new projects      0.88  + PASS
  Ask targeted, open-ended questions du...  0.88  + PASS
  Summarize progress and check direction    0.78  + PASS
  Recap accomplishments and confirm met...  0.83  + PASS
  Summarize session and suggest next st...  0.89  + PASS

  Avg time: 38.5s
======================================================================

Related Issue(s)

Type of Change

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with prompt-builder agent and addressed all feedback
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Testing

The workflow has been validated by triggering it manually via workflow_dispatch. All 30 evaluation cases passed with an overall score of 0.81.

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used /prompt-analyze to review contribution
• Addressed all feedback from prompt-builder review
• Verified contribution follows common standards and type-specific requirements

Required Automated Checks

• Markdown linting: npm run lint:md
• Spell checking: npm run spell-check
• Frontmatter validation: npm run lint:frontmatter
• Skill structure validation: npm run validate:skills
• Link validation: npm run lint:md-links
• PowerShell analysis: npm run lint:ps
• Plugin freshness: npm run plugin:generate

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues
• Security-related scripts follow the principle of least privilege

Additional Notes

All GitHub Actions uses: steps are pinned to SHA hashes per supply chain security best practices.

[eedorenko]

FYI @vyta @bjcmit

[WilliamBerryiii]

Thank you for this PR, @eedorenko. Behavioral evaluation for the dt-coach agent is a valuable addition, and we appreciate the effort to formalize agent quality testing with structured evaluation cases.

After reviewing the workflow changes against our CI security standards, we've identified several issues that need to be resolved before this can merge. The findings fall into two categories: supply-chain security violations in the beval workflow, and architectural concerns with integrating it into PR validation and release pipelines.

Important

The combination of unpinned dependencies from an external personal repository, unpinned npm range installs, inherited secrets, and persisted credentials creates a compound risk. A compromise of any one dependency effectively grants access to all repository secrets and the CI execution context.

We've added inline comments on each affected file with specific context and suggested changes. The critical items are:

• pip install from vyta/beval with no commit SHA and no hash verification (see comment on beval.yml line 32)
• npm install -g @github/copilot@1 with a major-version range and no lockfile (see comment on beval.yml line 29)
• actions/checkout without persist-credentials: false (see comment on beval.yml line 21)
• Both copilot instances launch with --allow-all, granting unrestricted permissions (see comment on beval.yml line 36)
• secrets: inherit in both calling workflows forwards all repository secrets when only COPILOT_TOKEN is needed
• Behavioral evaluation should not gate PR merges or releases at this stage (see comments on pr-validation.yml and release-stable.yml)

Our repository enforces these standards through Test-DependencyPinning.ps1, Test-WorkflowPermissions.ps1, and the conventions documented in workflows.instructions.md. The copilot-setup-steps.yml workflow demonstrates the expected pattern for downloading and verifying external binaries.

We recommend deploying beval as a standalone workflow_dispatch or scheduled workflow instead of integrating it into pr-validation.yml and release-stable.yml. This allows behavioral testing to proceed without gating contributor workflows or release processes.

Please comment if you have questions about any of the suggestions, and we can discuss further.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.89%. Comparing base (e69486a) to head (e2b0414).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1129      +/-   ##
==========================================
- Coverage   86.90%   86.89%   -0.02%     
==========================================
  Files          59       59              
  Lines        8774     8774              
==========================================
- Hits         7625     7624       -1     
- Misses       1149     1150       +1     

Flag	Coverage Δ
pester	85.32% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[katriendg]

Truly love this addition, adding un-deterministic evaluations to DT coach. This is an area we wanted to explore, thanks for adding this!

Given the very experimental status of beval, security hardening of the repo, and APIs subject to change without notice, we probably want to make this very experimental as well, and not have any hard dependencies on this workflow.
Could we see it as a run we can manually trigger at some occasions, not entirely sure what approach would be most valuable to learn from? I fully agree with @WilliamBerryiii that we don't want these un-deterministic/behavioral workflows to be part of PR reviews or even release validations, at least not for a while.

[WilliamBerryiii]

I've resolved the dependency pinning issues and license violation in the PR #1159 - this was related to dependent package licensing .. not the GHAS alert above.