ci: add promptfoo code scan action

[promptfoo-scanner]

Add Promptfoo Code Scan Action

This PR adds a GitHub Action workflow to automatically scan pull requests for LLM-related vulnerabilities.

What this does

• Runs on every pull request
• Scans code changes for potential security issues in LLM integrations
• Posts findings as PR comments

---

This PR was automatically created by the Promptfoo GitHub App

[greptile-apps]

Greptile Summary

This PR adds a new GitHub Actions workflow (.github/workflows/promptfoo-code-scan.yml) that automatically runs a Promptfoo LLM security scan on pull requests and posts findings as PR comments. The intent is sound, but the workflow has three issues that should be addressed before merging:\n\n- Missing synchronize trigger: The workflow only fires on opened and ready_for_review. Commits pushed to an already-open PR are never scanned, which defeats most of the security value.\n- Overly broad id-token: write permission: OIDC token write access is granted to a third-party action (promptfoo/code-scan-action@v0). This allows the action to request tokens that can authenticate with cloud providers, which is a significant privilege if not strictly required by the action.\n- Unpinned mutable action tag: @v0 is a mutable ref. Pinning to a full commit SHA is the recommended practice to guard against supply-chain attacks or accidental breaking changes.

Confidence Score: 2/5

Not safe to merge as-is due to a security permission concern and a functional gap in PR coverage.

Two P1 issues are present: the id-token: write permission granted to a third-party action is a meaningful supply-chain security risk, and the missing synchronize trigger means the scan is ineffective for the majority of PR activity (iterative commits). Both need to be addressed before this workflow provides reliable security value.

.github/workflows/promptfoo-code-scan.yml requires attention for all three flagged issues.

Important Files Changed

Filename	Overview
.github/workflows/promptfoo-code-scan.yml	New CI workflow that runs a promptfoo LLM security scan on PRs; has a missing synchronize trigger, an overly broad id-token: write permission granted to a third-party action, and an unpinned mutable action tag.

Sequence Diagram

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant Runner as GH Actions Runner
    participant PSA as promptfoo/code-scan-action@v0
    participant OIDC as GitHub OIDC Provider

    Dev->>GH: Opens PR (or marks ready_for_review)
    GH->>Runner: Trigger: pull_request [opened | ready_for_review]
    Note over GH,Runner: ⚠️ synchronize NOT in trigger list — new commits skipped
    Runner->>Runner: actions/checkout@v4 (full history)
    Runner->>OIDC: Request OIDC token (id-token: write)
    Note over Runner,OIDC: ⚠️ Token granted to third-party action
    OIDC-->>Runner: OIDC JWT
    Runner->>PSA: Run promptfoo/code-scan-action@v0
    Note over PSA: ⚠️ Mutable tag — version may change silently
    PSA-->>GH: Post findings as PR comment (pull-requests: write)

Loading

_{Reviews (1): Last reviewed commit: "Add Promptfoo Code Scan workflow" | Re-trigger Greptile}

[greptile-apps]

Scan misses commits pushed to open PRs

The trigger only fires on opened and ready_for_review, but not synchronize. This means any new commits pushed to an already-open PR will never be scanned, leaving a significant gap in coverage — the exact situation where a developer iterates on code after initial review.

Suggested change

	types: [opened, ready_for_review]
	types: [opened, synchronize, ready_for_review]

[greptile-apps]

Overly broad id-token: write permission granted to third-party action

id-token: write allows the job to request an OIDC token from GitHub, which can be used to authenticate with cloud providers (AWS, GCP, Azure). This permission is being granted to promptfoo/code-scan-action@v0, a third-party action. If the action (or its supply chain) is compromised, it could exfiltrate cloud credentials.

Unless the promptfoo action explicitly documents requiring OIDC authentication, this permission should be removed. If it is genuinely required, the action's documentation should be referenced in a comment.

[greptile-apps]

Action pinned to mutable tag, not a commit SHA

Using @v0 is a mutable tag — if the upstream repo updates the tag (or is compromised), this workflow will silently start running different code. GitHub's security hardening guide recommends pinning third-party actions to a full commit SHA:

Suggested change

	uses: promptfoo/code-scan-action@v0
	uses: promptfoo/code-scan-action@<full-commit-sha> # v0

Replace <full-commit-sha> with the exact SHA corresponding to the v0 release you've validated.