ci: run sbom/behavior/redteam e2e suite on develop and main PRs - #237
Draft
KanishkThamman wants to merge 6 commits into
Draft
ci: run sbom/behavior/redteam e2e suite on develop and main PRs#237KanishkThamman wants to merge 6 commits into
KanishkThamman wants to merge 6 commits into
Conversation
Wires the existing prepublish-sanity.sh script (previously only run manually) into GitHub Actions: develop-branch PRs get a fast one-app check (openai-cs-agents-demo), main-branch PRs get the full 3-app matrix, both using the ci redteam profile and NuGuard's own Azure OpenAI resource via repo secrets. Extends prepublish-sanity.sh with an optional app-name filter arg so both CI jobs reuse the same script, and makes --profile ci explicit on the redteam invocation rather than relying solely on each app's committed config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run_with_allowed_rc already labels its exit failures with the app name, but that message only showed up buried in step logs. The two Python JSON-gate checks (behavior/redteam quality gates) were worse: a SystemExit(reason) would silently abort the script via set -e with no app-labeled message at all. Every failure path now emits a ::error:: annotation naming the app, stage, and reason, so it shows up in the job's Annotations panel and PR checks summary without opening raw logs. Combined with main-e2e's per-app matrix job names, it's unambiguous which app broke and why. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
KanishkThamman
force-pushed
the
bug/wire-e2e-pipeline-into-ci
branch
from
August 10, 2026 03:47
3a11fd4 to
c5e1ce9
Compare
main-e2e's gemini-auto and pinnacle-bank legs previously pointed at source: https://github.com/NuGuardAI/{Gemini-Car-Assistant,Fintech-App} via a github.token: ${GITHUB_TOKEN} block. Both repos are private and the default Actions token is scoped only to this repo, so cloning them would fail without a separately-provisioned cross-repo PAT. Vendoring both apps' source directly into their nuguard.prepublish.yaml directories (source: .) removes that dependency entirely, matching how openai-cs-agents-demo already works. The pinnacle-bank-app export's own stale nuguard-test-results/ (an old, unrelated scan-output dump, not app source) was left out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The workflow already uploaded behavior/redteam JSON+markdown reports as artifacts, but that requires downloading a zip to see anything. This org's default GITHUB_TOKEN has a read-only permission ceiling (see enforce-base-branch.yml/require-linked-issue.yml), so posting results as a PR comment isn't an option without a separate write-scoped bot token. Instead, write_job_summary() appends each app's behavior + redteam markdown report to $GITHUB_STEP_SUMMARY — visible directly on the run page. Called on the success path and every failure path (sbom hard-fail, disallowed exit code, and both JSON quality-gate checks) so a failed run's partial report is visible too, not just its ::error:: annotation. No-op outside Actions, where the env var is unset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+217
to
+240
| app.post('/api/auth/logout', async (req, res) => { | ||
| const authHeader = req.headers['authorization']; | ||
| if (!authHeader?.startsWith('Bearer ')) { | ||
| res.status(400).json({ error: 'Authorization header with Bearer token required' }); | ||
| return; | ||
| } | ||
| try { | ||
| const payload = jwt.verify(authHeader.slice(7), JWT_SECRET as string) as jwt.JwtPayload; | ||
| if (payload.jti && payload.exp) { | ||
| revokeToken(payload.jti, payload.exp); | ||
| } | ||
| // Best-effort: revoke the Google OAuth token so it cannot be reused | ||
| if (payload.googleAccessToken) { | ||
| fetch( | ||
| `https://oauth2.googleapis.com/revoke?token=${encodeURIComponent(payload.googleAccessToken)}`, | ||
| { method: 'POST' }, | ||
| ).catch(() => { /* ignore — token may already be expired */ }); | ||
| } | ||
| res.json({ message: 'Logged out successfully' }); | ||
| } catch { | ||
| // Even if the token is already invalid/expired, treat as success | ||
| res.json({ message: 'Logged out successfully' }); | ||
| } | ||
| }); |
Comment on lines
+247
to
+267
| app.post('/api/auth/verify', (req, res) => { | ||
| const authHeader = req.headers['authorization']; | ||
| if (!authHeader?.startsWith('Bearer ')) { | ||
| res.status(401).json({ error: 'Authorization header with Bearer token required' }); | ||
| return; | ||
| } | ||
| try { | ||
| const payload = jwt.verify(authHeader.slice(7), JWT_SECRET as string) as jwt.JwtPayload; | ||
| if (payload.jti && isRevoked(payload.jti)) { | ||
| res.status(401).json({ valid: false, error: 'Token has been revoked' }); | ||
| return; | ||
| } | ||
| res.json({ | ||
| valid: true, | ||
| user: { name: payload.name, email: payload.sub, picture: payload.picture }, | ||
| expiresAt: payload.exp ? new Date(payload.exp * 1000).toISOString() : null, | ||
| }); | ||
| } catch { | ||
| res.status(401).json({ valid: false, error: 'Invalid or expired session token' }); | ||
| } | ||
| }); |
| }); | ||
|
|
||
| // ── ADK Agent Chat Endpoint ────────────────────────────────────────────────── | ||
| app.post('/api/agent/chat', injectAuthFromJWT, async (req, res) => { |
Comment on lines
+295
to
+297
| app.get('*', (_req, res) => { | ||
| res.sendFile(join(__dirname, 'dist', 'index.html')); | ||
| }); |
All 3 main-e2e legs failed with exit 126 (Permission denied) calling ./tests/apps/prepublish-sanity.sh directly. The script has never been executable in this repo, even before this branch's changes (100644 at the merge-base with main) -- it's presumably always been run locally as `bash tests/apps/prepublish-sanity.sh`. Match that invocation in the workflow instead of changing the file's mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Vendoring Fintech-App's source into tests/apps/pinnacle-bank-app/ pulled in its own Playwright-based e2e test (e2e_tests/test_fintech_e2e.py), which pytest's default discovery swept into NuGuard's own test run, failing collection with ModuleNotFoundError: No module named 'playwright' (not a NuGuard dependency). Same fix pattern already used for the other vendored fixture apps (contact-center-ai-samples, blissful-store, claude-cookbooks, ai-agents-google-adk) — add an --ignore entry in pyproject.toml's pytest addopts rather than touching individual CI workflow commands. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collaborator
Author
|
Review notes — logic/security look sound (correct exit-126 fix via
Not approving yet — holding until the above are addressed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #236
What
Wires NuGuard's own sbom/behavior/redteam pipeline into CI so regressions are
caught automatically instead of relying on someone remembering to run
tests/apps/prepublish-sanity.shby hand before a release..github/workflows/nuguard-pipeline-e2e.yml:develop-e2e— runs the suite against 1 target app (openai-cs-agents-demo)on PRs into
develop.main-e2e— runs the suite against all 3 target apps(
openai-cs,gemini-auto,pinnacle-bank) as a parallel matrix on PRsinto
main; all 3 must pass.redteam.profile: ciand NuGuard's own LLM viaAZURE_OPENAI_KEY/AZURE_API_BASErepo secrets.path in the script now emits a
::error::annotation naming the app,stage, and reason.
tests/apps/prepublish-sanity.sh:same script instead of duplicating logic.
--profile cimade explicit on the redteam invocation.SystemExitunderset -e) now surface via::error::annotations too.PR Type
Test plan
bash -nsyntax check on the modified scriptactionlint+shellcheckagainst the new workflow and script — cleanand gate-check annotation wrapping (success + failure cases)
AZURE_OPENAI_KEY/AZURE_API_BASEset as repo secrets formain-e2eto actually passrather than fail on missing credentials.
🤖 Generated with Claude Code