ai-ops-security-kit

A small, sharp toolkit for the security problems that show up specifically in AI / LLM work — the ones generic secret scanners and SAST tools miss. Five cross-platform scripts (Bash + PowerShell), no runtime dependencies, every tool covered by a reproducible test harness and CI.

Most "AI security" tooling is either a SaaS dashboard or a research paper. This is the opposite: drop-in scripts you can run in a pipeline or on a laptop in seconds.

---

Why this exists

The fastest-growing source of credential leaks and injection bugs right now is AI glue code — prompt templates, agent tool-wiring, provider keys sprayed across .env files and notebooks. The patterns are distinct from classic appsec:

• provider keys (sk-…, sk-ant-…, hf_…) that generic scanners don't fingerprint,
• model output flowing into shells, templates and eval sinks,
• local inference servers (Ollama, LM Studio, vLLM) exposed over plaintext HTTP,
• token spend quietly 10×-ing overnight.

This kit targets exactly those.

Tools

Tool	Lang	What it does
bash/scan-api-keys.sh	Bash	Fingerprints AI + cloud provider keys; entropy-gates generic matches to cut false positives. text/json output, allowlist support.
bash/prompt-injection-lint.sh	Bash	Static linter for prompt-injection / tool-abuse patterns (unsanitized interpolation, output→shell sinks, tool calling without allowlist, LLM→HTML XSS).
powershell/Get-LLMKeyExposure.ps1	PowerShell	Key-exposure scan across files and the live environment (env vars) — the dev-box leak vector.
powershell/Test-AIEndpointHealth.ps1	PowerShell	Reachability / latency / TLS posture for local + cloud inference endpoints. Sends no credentials or prompts; flags plaintext HTTP.
powershell/Measure-TokenCost.ps1	PowerShell	Audits a JSONL usage log, costs it against a configurable price table, and flags anomalous spend days (mean + σ).

Usage

# Scan a repo for leaked AI/cloud keys (exit 1 if any found — CI-friendly)
./bash/scan-api-keys.sh ./src
./bash/scan-api-keys.sh -f json . > findings.json

# Lint prompt/agent code for injection risk, high severity only
./bash/prompt-injection-lint.sh -s high ./app

# Find exposed keys in files + environment
./powershell/Get-LLMKeyExposure.ps1 -Path ./src -AsJson

# Check which inference endpoints are up, fast, and TLS-secured
./powershell/Test-AIEndpointHealth.ps1

# Audit token spend and surface spike days
./powershell/Measure-TokenCost.ps1 -LogPath ./usage.jsonl -Sigma 1.5

Install

git clone https://github.com/aharwelik/ai-ops-security-kit.git
cd ai-ops-security-kit
chmod +x bash/*.sh

No packages to install. Bash tools use only POSIX utilities + awk; PowerShell tools are pure pwsh.

Requirements

• Bash 3.2+ (runs on stock macOS bash and modern Linux — no mapfile/associative-array dependencies)
• PowerShell 7+ (pwsh, cross-platform)
• awk, grep, find (present on macOS/Linux)

Testing

Every tool has assertions in a self-contained harness, and CI runs it on every push:

bash tests/run-tests.sh

Verified locally on macOS (bash 3.2.57, PowerShell 7.4.6) — 11 passed, 0 failed.

Scope & ethics

Defensive tooling only — auditing, detection, and hardening. The scanners are heuristic: a clean run reduces risk but is not a proof of safety. Validate findings before acting, and never commit real secrets to test fixtures (the bundled fixtures use obvious placeholders).

Author

Anthony Harwelik — security & AI engineering aharwelik@gmail.com · github.com/aharwelik

_{© 2026 Anthony Harwelik · MIT License · Built to be run, not just read.}