From ddd7567696b88e3a93c6ca8936ac45e39e9e9cf0 Mon Sep 17 00:00:00 2001 From: BB-AI-Arena <85533977+BB-AI-Arena@users.noreply.github.com> Date: Thu, 13 Aug 2026 19:08:44 -0500 Subject: [PATCH] expand project vision and use cases --- CHANGELOG.md | 7 ++ README.md | 77 ++++++++++++++++++++- docs/PROJECT_VISION.md | 150 +++++++++++++++++++++++++++++++++++++++++ docs/USE_CASES.md | 135 +++++++++++++++++++++++++++++++++++++ 4 files changed, 368 insertions(+), 1 deletion(-) create mode 100644 docs/PROJECT_VISION.md create mode 100644 docs/USE_CASES.md diff --git a/CHANGELOG.md b/CHANGELOG.md index d56b056..716554c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ All notable changes to this project will be documented here. +## Unreleased + +### Documentation + +- Expanded the project narrative, capability reference, applications, outcomes, and production path +- Added detailed project-importance and real-world use-case guides + ## [0.4.0] - 2026-08-12 ### Added diff --git a/README.md b/README.md index 1c0ad0c..4e75268 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ **A context-aware command execution gate that asks one critical question before code runs: _does this action match the user's intent?_** -[Quick start](#quick-start) · [How it works](#how-it-works) · [Integrations](#security-integrations) · [Dashboard](#grafana-dashboard) · [Security](SECURITY.md) +[Why it matters](#why-this-project-matters) · [Capabilities](#what-the-project-can-do) · [Use cases](#where-it-can-be-used) · [Quick start](#quick-start) · [Architecture](docs/ARCHITECTURE.md) · [Security](SECURITY.md) @@ -39,6 +39,70 @@ User Intent AI Security adds a pre-execution decision layer. It combines command The ordinary-command hot path is local and deterministic. External APIs, SIEM queries, provenance scans, and manager notifications run asynchronously and contribute cached signals without delaying execution. +## Why this project matters + +Modern command execution is no longer limited to a person carefully typing into a terminal. Commands are proposed and launched by AI coding assistants, autonomous agents, deployment systems, remote-management tools, scripts, and privileged operators working under time pressure. These systems can produce syntactically valid commands that are still wrong for the task, wrong for the target, or dangerous in the current security context. + +Most controls answer one of four questions: _is the file malicious, is the identity authorized, is the command on a denylist, or did something suspicious already happen?_ User Intent AI Security explores a complementary question **before process creation**: + +> **Given what this user or agent appears to be trying to accomplish, should this exact command be allowed to run here, now, with this privilege and blast radius?** + +That distinction is important because many damaging actions use legitimate, signed, administrator-approved tools. `rm`, PowerShell, `terraform`, `kubectl`, cloud CLIs, database clients, and Git are indispensable—and can also erase data, disable controls, expose secrets, or change production in seconds. Intent-aware enforcement creates a place to combine the command, the task, recent behavior, privilege, project health, and live security posture into one explainable decision. + +This project is intended to help teams investigate a missing security layer between **authorization** and **execution**. It does not replace EDR, SIEM, PAM, WAF, or change management; it gives those systems a chance to influence a decision before an irreversible action starts. Read the full [Project Vision](docs/PROJECT_VISION.md). + +## What the project can do + +| Capability | What it evaluates | Why it is useful | +|---|---|---| +| Intent-aware command gating | Command text, optional stated purpose, recent commands, and local context | Finds risky actions that do not fit the declared task | +| Three-way policy decision | `ALLOW`, `REVIEW`, or `BLOCK` with a 0–100 risk score | Supports silent low-risk work, human confirmation, and hard stops | +| Blast-radius analysis | Recursive deletion, broad targets, infrastructure destruction, cloud deletion, and other irreversible patterns | Separates a routine change from an operation with widespread impact | +| Privilege amplification | Linux root and Windows elevated administrator context | Applies greater scrutiny when the same command can do more damage | +| Behavioral anomaly detection | Per-user command-family frequency from prior gated activity | Highlights unusual administration without treating novelty alone as guilt | +| Sequence awareness | Recent sensitive actions, such as secret access followed by network transfer | Detects risk that becomes visible only across multiple commands | +| Project and release context | Git state, tests, code-health indicators, and cached provenance scan | Adds friction before publishing or deploying suspicious or unfinished code | +| Security posture correlation | Cached AV, EDR, SIEM, device, user, and project-scoped signals | Tightens command policy during an active endpoint or identity incident | +| Cross-platform destructive catalog | Windows, Linux, macOS, containers, databases, Kubernetes, cloud, Git, and recovery controls | Provides a transparent starting policy that teams can inspect and extend | +| Explainable evidence | Named signals, score contributions, command fingerprint, and latency | Lets operators understand why a decision occurred and tune policy responsibly | +| Privacy-conscious escalation | Secret redaction, local queueing, thresholds, and asynchronous webhook delivery | Enables human oversight without placing notification latency in the command path | +| Operational visibility | Prometheus metrics and a provisioned Grafana dashboard | Exposes decision volume, latency, posture, sources, and report backlog | +| Protected integration API | OWASP CRS WAF, backend network isolation, bearer-token ingestion, and bounded requests | Reduces attack surface for the security signals that influence policy | +| Repeatable deployment | Docker Compose, Terraform, and Ansible | Makes the POC reproducible for labs, demos, and controlled evaluations | + +## Where it can be used + +The wrapper is the demonstration surface; the policy model can sit at any trusted point that owns process creation or privileged actions. + +| Environment | Example use | +|---|---| +| AI coding agents | Evaluate every shell/tool call against the user's requested task before execution | +| Developer workstations | Review destructive commands, force pushes, package publication, and risky download-to-shell pipelines | +| Privileged access workstations | Increase scrutiny for root or administrator actions and unusual operational sequences | +| CI/CD systems | Gate deploy, publish, infrastructure-apply, and rollback commands using pipeline and security context | +| Server administration | Wrap SSH command brokers, remote-management agents, or endpoint services that create processes | +| Kubernetes and cloud operations | Add intent, scope, environment, and incident posture to destructive control-plane operations | +| Database operations | Require review or approval for destructive SQL and unusually broad maintenance actions | +| Security automation | Prevent a compromised automation identity from using legitimate tools outside its expected workflow | +| Training and purple-team labs | Demonstrate how benign tools become dangerous when purpose, sequence, privilege, and posture change | + +Detailed walkthroughs—including accidental deletion, compromised admin sessions, AI-agent drift, deployment protection, and incident-aware policy—are in [Use Cases and Scenarios](docs/USE_CASES.md). + +## What a decision looks like + +The same executable can receive a different outcome as context changes: + +| Situation | Likely result | Contributing context | +|---|---|---| +| Developer runs `git status` | `ALLOW` | Common read-only action | +| Developer force-pushes reviewed history | `REVIEW` | External side effect and history rewrite | +| Administrator deletes recovery snapshots | `BLOCK` | Destructive recovery action plus elevated privilege | +| Agent publishes from a dirty, untested project | `REVIEW` or `BLOCK` | Publish action, Git state, missing tests, and provenance risk | +| User transfers data after accessing secrets | `BLOCK` | Sequence risk and possible exfiltration | +| Routine admin command during a critical EDR alert | Stricter than normal | Privilege combined with correlated endpoint posture | + +Every result includes named reasons and score contributions. The intent is not to make an opaque model the final authority; it is to create an auditable policy decision that can include deterministic rules, cached analytical signals, and human review. + ## Highlights - **Sub-millisecond policy evaluation** — approximately 0.53 ms for context collection and scoring in the development benchmark. @@ -281,6 +345,17 @@ Version **0.4.0** is a research-quality proof of concept. Important production w - Evaluate anomaly quality against representative benign and adversarial datasets. - Add durable encrypted storage and enterprise identity for reports and policy administration. +### What this POC proves today + +- A useful decision can be produced without placing network or model calls in the execution hot path. +- Legitimate tools can be evaluated by purpose, sequence, privilege, scope, project state, and external posture—not only by binary reputation. +- AV, EDR, SIEM, WAF, audit, reporting, and user-context signals can be normalized around a single pre-execution decision. +- The decision can remain explainable enough for testing, policy tuning, and human review. + +### Path toward production + +The next stage is a protected, persistent command broker with shell-specific parsing, resolved resource targets, signed policies, enterprise identity, approval workflows, tamper-resistant storage, and measured false-positive/false-negative performance. See the [Project Vision](docs/PROJECT_VISION.md#roadmap-from-poc-to-production) for the proposed phases and success criteria. + ## Responsible use This project observes commands and can generate workplace security reports. Deploy it transparently, collect only what is necessary, protect the resulting data, and provide meaningful human review. Do not treat anomaly scores or AI-related signals as proof of malicious intent. diff --git a/docs/PROJECT_VISION.md b/docs/PROJECT_VISION.md new file mode 100644 index 0000000..c290f75 --- /dev/null +++ b/docs/PROJECT_VISION.md @@ -0,0 +1,150 @@ +# Project vision: security between authorization and execution + +## Executive summary + +User Intent AI Security explores a pre-execution control for human and machine-generated commands. Its central idea is simple: authorization to use a tool does not imply that every possible use of that tool is appropriate. Before a process starts, a trusted command broker should evaluate whether the requested action matches the user's stated objective, recent activity, normal behavior, privilege, target scope, project condition, and current security posture. + +The project is deliberately broader than a command denylist. It is a reference architecture for combining deterministic safety policy, behavioral context, software provenance indicators, security-product signals, explainable scoring, human review, and operational telemetry without adding vendor or model latency to routine commands. + +## The security gap + +Organizations already invest in identity, endpoint protection, logging, network security, privileged access, and change controls. Those systems are essential, but they frequently operate before or after the point where intent becomes an operating-system action: + +- Identity and PAM establish who may act, but not whether a specific action matches the current task. +- Allowlisting establishes which tools may run, but powerful approved tools can perform harmful operations. +- EDR can detect suspicious behavior, but detection may occur after a process has started or damage has begun. +- SIEM correlates events well, but it is usually outside the synchronous command path. +- Change systems document expected work, but the terminal does not automatically compare each command with that expectation. +- AI safety filters reason about generated text, but the operating system ultimately executes concrete commands with real privileges and targets. + +An intent gate connects these layers at the last responsible moment: after a command has been proposed, but before it executes. + +## Why the problem is becoming more urgent + +### AI agents increase command volume and speed + +An AI agent can produce and execute many commands faster than a person can review them. Even when its high-level objective is correct, it may misunderstand the environment, choose an overly broad target, continue from stale context, or turn an untrusted response into executable input. A low-friction policy boundary lets an organization supervise actions without requiring a person to approve every harmless read-only command. + +### Legitimate tools are dual-use + +Administrative utilities, shells, infrastructure tools, database clients, and cloud CLIs are normal components of enterprise work. Malware reputation cannot distinguish a carefully scoped maintenance command from the same trusted binary deleting production resources or weakening security controls. + +### Privilege makes ordinary mistakes exceptional + +A typo in an unprivileged sandbox and the same typo as root have different consequences. Intent-aware policy can treat privilege as a risk amplifier instead of assuming that successful elevation settles the safety question. + +### Security context is fragmented + +An endpoint alert, unusual identity event, dirty repository, missing tests, secret-access sequence, and destructive command may each be inconclusive alone. Together they can justify review or prevention. The gate provides a common decision point for those otherwise disconnected facts. + +## Core principles + +1. **Decide before execution.** Prevention must occur at a process-creation or privileged-action boundary, not only in downstream logs. +2. **Keep the common path local.** Routine decisions should not wait on a model, SaaS API, SIEM query, or webhook. +3. **Treat context as time-bound evidence.** External signals expire, behavioral baselines evolve, and stale information should not silently remain authoritative. +4. **Make decisions explainable.** Operators need named signals, score contributions, policy versions, and reproducible outcomes. +5. **Use privilege and blast radius as multipliers.** Risk depends on what the action can affect, not only on command keywords. +6. **Preserve human authority for ambiguity.** `REVIEW` is a first-class outcome between silent execution and hard denial. +7. **Minimize and protect telemetry.** Commands and reports can contain secrets or sensitive employee activity; collection, redaction, access, and retention must be deliberate. +8. **Do not equate anomaly with malice.** Novel behavior is a reason to add context, not proof of wrongdoing. +9. **Fail deliberately.** Feed outages, policy errors, and broker failures need explicit fail-open, fail-closed, or review-only behavior by action class. +10. **Assume the wrapper can be bypassed.** Production value requires enforcement in the component that actually owns execution. + +## Strategic value + +### For security teams + +- Convert endpoint and SIEM findings into immediate, scoped execution policy. +- Interrupt defense evasion, recovery deletion, suspicious exfiltration sequences, and high-risk administration earlier. +- Produce structured, explainable evidence for investigation instead of an unexplained block. +- Test policy changes in review or detection mode before enforcement. + +### For platform and operations teams + +- Add safety checks to deployment, cloud, Kubernetes, database, and infrastructure workflows. +- Distinguish normal automation from commands outside a service account's expected pattern. +- Reduce the chance that a typo, stale runbook, or wrong environment becomes an outage. +- Route ambiguous operations into approval instead of banning powerful tools entirely. + +### For AI engineering teams + +- Put a deterministic control beneath model-generated tool calls. +- Compare concrete actions with the user's original objective and recent agent trajectory. +- Keep security enforcement independent of which model or agent framework proposed the command. +- Gather decision telemetry for evaluating agent safety without treating model confidence as authorization. + +### For governance and risk teams + +- Define auditable policy around sensitive actions and privileged context. +- Measure how often risky commands are allowed, reviewed, blocked, overridden, or associated with active incidents. +- Establish transparent employee notice, retention, access, appeal, and oversight practices for behavioral signals. + +## Reference operating model + +```mermaid +flowchart LR + R["Requested task"] --> A["Human or AI agent"] + A --> B["Trusted command broker"] + B --> C["Local context + policy"] + E["Cached EDR / SIEM / identity posture"] --> C + P["Signed policy + approved exceptions"] --> C + C -->|"ALLOW"| X["Process creation"] + C -->|"REVIEW"| H["Human or workflow approval"] + C -->|"BLOCK"| D["Deny + explain"] + H -->|"Approved"| X + C --> T["Audit + metrics"] + T --> S["Security operations"] +``` + +The broker is the enforcement boundary. It receives structured arguments, user and device identity, a task or change reference, working context, and cached security posture. A versioned policy produces an outcome. High-confidence safe actions proceed immediately; ambiguous actions use an approval workflow; dangerous actions stop before process creation. + +## What success should look like + +A production implementation should be evaluated with more than raw block counts: + +| Objective | Example measure | +|---|---| +| Invisible normal operation | p50/p95/p99 decision latency and percentage of routine commands silently allowed | +| Prevention quality | Confirmed harmful or out-of-intent actions blocked before execution | +| Operator usability | Review rate, approval time, override rate, and abandonment rate | +| Detection quality | False-positive and false-negative rates on representative workflows | +| Explainability | Percentage of decisions with actionable, policy-linked reasons | +| Resilience | Behavior during feed, storage, policy, and broker failures | +| Privacy | Data fields collected, redaction effectiveness, retention, and access audit coverage | +| Policy health | Drift, exception age, unused rules, and decision changes between policy versions | + +## Roadmap from POC to production + +### Phase 1: research and evaluation + +- Expand benign and adversarial command datasets. +- Benchmark latency and detection quality across Windows and Linux. +- Tune the destructive catalog and external signal normalization. +- Validate privacy controls and reporting workflows with stakeholders. + +### Phase 2: trusted command broker + +- Move enforcement from an optional CLI wrapper into a persistent service that owns process creation. +- Accept structured executable and argument arrays rather than relying on reconstructed command strings. +- Add PowerShell, POSIX shell, and platform-specific parsers. +- Resolve filesystem, database, cloud, cluster, and infrastructure targets before scoring blast radius. + +### Phase 3: enterprise policy and identity + +- Add signed policy bundles, versioning, staged rollout, simulation, and rollback. +- Integrate device identity, workforce identity, PAM, ticket/change context, and approval systems. +- Protect baselines, reports, and exceptions with encryption and role-based access. +- Define feed-health behavior per command class. + +### Phase 4: production learning and assurance + +- Continuously measure false positives, false negatives, overrides, and user friction. +- Add adversarial testing for parser confusion, baseline poisoning, stale context, and policy bypass. +- Support fleet management, attestation, tamper evidence, and independent audit export. +- Develop organization-specific policies without losing a portable, inspectable core. + +## Boundaries and responsible use + +This project must not be presented as proof that a user is malicious or that code was authored by AI. Behavioral rarity, code-health markers, and correlated alerts are fallible signals. High-impact decisions need transparent policy, meaningful review, protected evidence, and a way to challenge mistakes. + +The current CLI is a proof of concept and is bypassable. The WAF protects the HTTP integration surface, not local process creation. Production deployment requires additional engineering, threat modeling, testing, legal and privacy review, and integration at a trusted execution boundary. diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md new file mode 100644 index 0000000..99eb1c8 --- /dev/null +++ b/docs/USE_CASES.md @@ -0,0 +1,135 @@ +# Use cases and scenarios + +This guide illustrates how an intent-aware command gate can be applied. Outcomes are examples; actual decisions depend on policy, context, thresholds, and the quality of integrated signals. + +## 1. Prevent an accidental broad deletion + +**Situation:** An operator intends to remove one generated build directory but issues a recursive deletion against a parent or root-level target. + +**Signals:** destructive filesystem action, broad target, elevated privilege, missing or mismatched purpose. + +**Possible outcome:** `BLOCK` before process creation, with an explanation identifying both the destructive action and blast-radius concern. + +**Value:** The control focuses on consequence and scope. It does not ban normal cleanup commands; it adds friction when the target makes the action difficult to recover from. + +## 2. Supervise an AI coding agent + +**Situation:** A user asks an agent to run tests. After encountering an error, the agent proposes disabling a security control, downloading a script into a shell, or force-pushing generated changes. + +**Signals:** declared-purpose mismatch, download-to-execution, obfuscation, security-control change, publish action, dirty worktree, or missing tests. + +**Possible outcome:** Harmless inspection and test commands remain silent; the divergent operation is sent to `REVIEW` or `BLOCK`. + +**Value:** The safety boundary evaluates the concrete tool call independently of the model that produced it. This supports multiple agent frameworks without trusting each agent to police itself. + +## 3. Tighten policy during an endpoint incident + +**Situation:** EDR reports credential-theft behavior on a device. An administrator session then attempts an unusual account, firewall, or persistence change. + +**Signals:** critical external posture, elevated privilege, behavioral anomaly, destructive or persistence action. + +**Possible outcome:** An operation that might normally require review crosses the blocking threshold while the alert is active. The signal expires after its TTL rather than affecting the device forever. + +**Value:** Existing security intelligence influences execution in near real time instead of remaining only in an analyst console. + +## 4. Protect production deployments + +**Situation:** A developer or pipeline runs `terraform apply`, `kubectl apply`, publishes a package, or pushes an image from a dirty or apparently untested project. + +**Signals:** external side effect, uncommitted changes, absent tests, cached provenance risk, unusual command family. + +**Possible outcome:** `REVIEW` asks for an explicit purpose or approval before the external change proceeds. + +**Value:** The gate creates a lightweight bridge between repository condition and release action. A production implementation could add environment, change-ticket, reviewer, and artifact-signature context. + +## 5. Detect a suspicious command sequence + +**Situation:** A session accesses likely credential material and then invokes a network-transfer tool. + +**Signals:** secret access, possible exfiltration, recent-command sequence risk, privilege, and external identity or endpoint alerts. + +**Possible outcome:** `BLOCK` or high-priority review with a redacted report for security operations. + +**Value:** Individual commands can look legitimate in isolation. Sequence awareness captures the relationship without requiring network calls in the command path. + +## 6. Reduce privileged administration mistakes + +**Situation:** An administrator uses routine service, account, firewall, disk, or recovery tooling from an elevated shell. + +**Signals:** root/admin context amplifies the score only when the command already has meaningful risk. + +**Possible outcome:** Read-only diagnostics remain quiet; service deletion, disk changes, recovery removal, or log erasure receive greater scrutiny. + +**Value:** Privilege is treated as consequence amplification, avoiding a blanket prompt on every administrative command. + +## 7. Constrain automation identities + +**Situation:** A CI runner or service account that normally builds artifacts begins changing identities, deleting cloud resources, or contacting an unexpected execution path. + +**Signals:** behavioral rarity, command family, action class, target scope, pipeline purpose, and SIEM/identity posture. + +**Possible outcome:** The operation is denied or routed to an approval workflow even though the service account is technically authorized. + +**Value:** Authorization establishes maximum capability; intent-aware policy constrains how that capability is expected to be used in a particular workflow. + +## 8. Guard database and infrastructure operations + +**Situation:** A maintenance task includes `DROP`, `TRUNCATE`, bulk deletion, cluster deletion, Terraform destruction, or cloud resource termination. + +**Signals:** destructive action class, resolved environment and resource scope, privilege, change reference, time window, and recent behavior. + +**Possible outcome:** Development resources may be reviewed while broad production targets are blocked without a verified approval. + +**Value:** A future target resolver can reason about the difference between one disposable resource and a production fleet rather than relying only on command keywords. + +## 9. Create an explainable security feedback loop + +**Situation:** Security engineering wants to understand which actions generate friction and which policy signals add value. + +**Signals and outputs:** named score contributions, decision latency, decision counts, source posture, overrides, and queued reports. + +**Possible outcome:** Grafana and exported audit data reveal false-positive hotspots, stale integrations, frequently reviewed actions, and policy gaps. + +**Value:** Teams can tune a visible policy system instead of operating an opaque blocker. Production evaluation should add policy versions, approval outcomes, and privacy-reviewed retention. + +## Deployment patterns + +### Developer or research workstation + +Use the CLI wrapper and local state to evaluate policy behavior. This is easy to test but bypassable, so it is appropriate for research and voluntary safety workflows. + +### Persistent endpoint broker + +Run a protected service that receives structured execution requests and is the only component allowed to launch managed commands. This is the preferred direction for endpoint enforcement. + +### AI tool gateway + +Place the gate between an agent runtime and its shell, file, deployment, or infrastructure tools. Include the user's task, agent step, tool arguments, workspace identity, and approval context. + +### CI/CD policy step + +Evaluate external-side-effect commands at a protected pipeline stage. Correlate branch protection, artifact provenance, environment, reviewer, change window, and security posture. + +### Privileged access integration + +Embed the policy in a PAM session broker, SSH gateway, remote-management service, or just-in-time elevation workflow so bypass requires crossing a separately controlled boundary. + +## Context worth adding in a production integration + +- Structured executable and argument list +- Resolved targets and environment classification +- User, workload, and device identity +- Original task, ticket, or change request +- Approval state and separation-of-duties requirements +- Data sensitivity and resource criticality +- Current endpoint, identity, cloud, and network posture +- Policy version and applicable exception +- Feed freshness and enforcement failure mode + +## What the project should not be used for + +- Inferring malicious intent from behavioral rarity alone +- Claiming reliable authorship detection for AI-generated code +- Secretly monitoring employees without notice, governance, access controls, and retention limits +- Replacing endpoint protection, identity controls, backups, testing, or change management +- Treating the current wrapper as a tamper-proof production boundary