diff --git a/concepts/defense/guard.mdx b/concepts/defense/guard.mdx index d5d55f0..bd5fec1 100644 --- a/concepts/defense/guard.mdx +++ b/concepts/defense/guard.mdx @@ -17,6 +17,10 @@ A Guard belongs to one of the following categories: Guards consist of a set of [Detectors](/concepts/evaluation-components/detector) and how they should be executed. + +The Vijil Console currently exposes Security, Moderation, and Privacy Guards in the **Protect** view. Integrity and Generic are broader Dome categories and do not appear as predefined Console Guard types. + + ## Next Steps @@ -26,8 +30,7 @@ Guards consist of a set of [Detectors](/concepts/evaluation-components/detector) Set up Guards for your agent - - Reference of all Guard types + + Review Guard types in the Console - diff --git a/concepts/defense/guardrail.mdx b/concepts/defense/guardrail.mdx index 00af4c7..535b583 100644 --- a/concepts/defense/guardrail.mdx +++ b/concepts/defense/guardrail.mdx @@ -12,6 +12,10 @@ Vijil Dome allows users to assemble and configure Guardrails, which are designed - **Retrieval Guardrails** (coming soon): To protect requests to and from retrievers. - **Execution Guardrails** (coming soon): To protect requests to and from external agents and tools. + +The Vijil Console currently configures Input and Output Guards from the selected Agent's **Protect** view. Retrieval and Execution Guardrails are broader concepts and are not sections in the current Console. + + ```mermaid actions={false} %%{init: {'theme':'base', 'themeVariables': {'fontFamily':'Futura Medium, Futura, sans-serif','fontSize':'13px'}, 'flowchart': {'nodeSpacing':25,'rankSpacing':35,'padding':6}}}%% flowchart TD @@ -95,7 +99,7 @@ flowchart LR ### Example Configuration -Here is an example of how you can set up Guards and Detectors (see the [Configuring Dome section](/tutorials/protect-agents/configuring-guardrails) for more details): +Here is an example based on the predefined Console Guards and Detectors (see the [Configuring Dome section](/owner-guide/protect-in-production/configuring-guardrails) for more details): ```python title="Python" icon="python" @@ -105,31 +109,31 @@ Here is an example of how you can set up Guards and Detectors (see the [Configur # Setup Guardrails from Guards ######################## # Input Guardrail - "input-guards": ["prompt-injection", "input-privacy"], + "input-guards": ["security-guard", "moderation-guard"], - # output guardrail - "output-guards": ["output-toxicity"], + # Output Guardrail + "output-guards": ["moderation-guard", "privacy-guard"], ########################## # Assemble and configure Guards ########################## - # a guard for prompt injection - "prompt-injection": { + # Security input Guard + "security-guard": { "type": "security", - "methods" : ["prompt-injection-deberta-v3-base", "security-llm"], + "methods": ["encoding-heuristics", "prompt-injection-mbert"], }, - # a guard to remove PII from requests to the LLM - "input-privacy": { - "type": "privacy", - "methods": ["privacy-presidio"] + # Moderation input and output Guard + "moderation-guard": { + "type": "moderation", + "methods": ["moderation-flashtext", "moderation-mbert"], }, - # a guard for toxic output content - "output-toxicity": { - "type": "moderation", - "methods": ["moderation-llamaguard"] + # Privacy output Guard + "privacy-guard": { + "type": "privacy", + "methods": ["privacy-presidio"], }, } ``` diff --git a/concepts/platform/console.mdx b/concepts/platform/console.mdx index dbbb784..8206e3c 100644 --- a/concepts/platform/console.mdx +++ b/concepts/platform/console.mdx @@ -53,4 +53,4 @@ Trust Reports are not just for engineers—they are designed for multiple stakeh - **Product and business leaders** use them to decide whether an agent is ready for deployment - **Stakeholders and executives** review them for final sign-off -Because the reports are standardized and easy to understand, they help overcome the gap between technical evaluation and decision-making. They provide evidence needed to move agents out of “_security limbo_” and into production. \ No newline at end of file +Because the reports are standardized and easy to understand, they help overcome the gap between technical evaluation and decision-making. They provide evidence needed to move agents out of “_security limbo_” and into production. diff --git a/concepts/welcome.mdx b/concepts/welcome.mdx index 16742aa..1e5449a 100644 --- a/concepts/welcome.mdx +++ b/concepts/welcome.mdx @@ -4,18 +4,18 @@ description: 'The trust layer for AI agents measure reliability, security, and s --- -**TL;DR:** Vijil is the trust layer for [AI agents](/owner-guide/register-agents/what-is-an-agent), measuring [reliability](/concepts/trust-score/reliability), [security](/concepts/trust-score/security), and [safety](/concepts/trust-score/safety) before deployment (Diamond) and protecting agents at runtime (Dome). Start with [Concepts](/concepts/trust-score/introduction) for the mental model, or jump to the [Owner's Guide](/owner-guide/getting-started/introduction) or [Developer Guide](/developer-guide/getting-started/introduction) to get started. +**TL;DR:** Vijil is the trust layer for [AI agents](/owner-guide/register-agents/what-is-an-agent), measuring [reliability](/concepts/trust-score/reliability), [security](/concepts/trust-score/security), and [safety](/concepts/trust-score/safety) before deployment (Diamond) and protecting Agents at runtime (Dome). Start with [Concepts](/concepts/trust-score/introduction) for the mental model, or jump to the [Owner's Guide](/owner-guide/getting-started/introduction) or [Developer Guide](/developer-guide/getting-started/introduction) to get started. Agents often stall before production: security and compliance need **proof**, not demos. Vijil turns that into **measurable trust**, systematic evaluation, runtime protection, and a feedback loop from real behavior. -**[Trust Score](/concepts/trust-score/introduction)** summarizes how an agent performs across **[reliability](/concepts/trust-score/reliability)**, **[security](/concepts/trust-score/security)**, and **[safety](/concepts/trust-score/safety)**; so reviews can reference evidence instead of opinions. Under that umbrella, **[Diamond](/concepts/platform/diamond)** probes behavior in depth through Trust Score evaluations and adaptive [Red Team](/concepts/platform/diamond#red-team-in-diamond) campaigns, while **Dome** enforces **[Guardrails](/concepts/defense/guardrail)** at runtime. +**[Trust Score](/concepts/trust-score/introduction)** summarizes how an Agent performs across **[reliability](/concepts/trust-score/reliability)**, **[security](/concepts/trust-score/security)**, and **[safety](/concepts/trust-score/safety)**; so reviews can reference evidence instead of opinions. Under that umbrella, **[Diamond](/concepts/platform/diamond)** probes behavior in depth through Trust Score evaluations and adaptive [Red Team](/concepts/platform/diamond#red-team-in-diamond) campaigns, while **Dome** enforces **[Guardrails](/concepts/defense/guardrail)** at runtime. ## How Vijil Measures Trust | Dimension | Question | Learn more | |-----------|----------|------------| -| **Reliability** | Does the agent do the right thing, consistently, under stress? | [Reliability](/concepts/trust-score/reliability) | +| **Reliability** | Does the Agent do the right thing, consistently, under stress? | [Reliability](/concepts/trust-score/reliability) | | **Security** | Can it resist abuse and protect sensitive data? | [Security](/concepts/trust-score/security) | | **Safety** | Does it stay within policy and avoid harmful outcomes? | [Safety](/concepts/trust-score/safety) | @@ -31,9 +31,9 @@ fast or ship resilient. production - Shipping fast increases failure rates and elevates risk; without -objective trust evidence, agents are in security limbo +objective trust evidence, Agents are in security limbo -- Generic red-teaming and [Guardrail](/concepts/defense/guardrail) do not ensure agents are +- Generic red-teaming and [Guardrail](/concepts/defense/guardrail) do not ensure Agents are safe for their specific context - Failure insights do not feed back into development, leaving the @@ -47,7 +47,7 @@ business context and personas. Turns 'is this safe?' into an answerable question with an audit-ready Trust Score. **Vijil Dome** -Enforces policies to protect and monitor agents at runtime +Enforces policies to protect and monitor Agents at runtime through embedded Guardrails, blocking attacks in real-time and logging detailed telemetry. @@ -62,7 +62,7 @@ Pick where you work and what you need next: **Start here** → [Trust Score](/concepts/trust-score/introduction), then explore this tab for deeper topics and the [Glossary](/concepts/glossary). - You use the Vijil console to [register agents](/owner-guide/register-agents/registering-agents), [run evaluations](/owner-guide/run-evaluations/running-evaluations), configure Dome, and report to stakeholders. + You use the Vijil console to [register Agents](/owner-guide/register-agents/registering-agents), [run Evaluations](/owner-guide/run-evaluations/running-evaluations), [configure Dome](/owner-guide/protect-in-production/configuring-guardrails), [monitor Dome Metrics](/owner-guide/protect-in-production/observability), and report to stakeholders. **Start here** → [Get started](/owner-guide/getting-started/introduction) in the console. @@ -77,6 +77,7 @@ Pick where you work and what you need next: - **Evaluate in the UI** → [Get started](/owner-guide/getting-started/introduction) - **Protect in production** → [Configuring Guardrails](/owner-guide/protect-in-production/configuring-guardrails) +- **Monitor protection** → [Dome Metrics](/owner-guide/protect-in-production/observability) ## Coming Soon @@ -84,9 +85,9 @@ Two additional products are currently in development and will expand the Vijil p - A catalog of hardened models and building blocks, including [Guardrail](/concepts/defense/guardrail) models tuned for agent safety, hardened LLMs optimized for specific tasks, and pre-validated components that reduce months of security work to days. + A catalog of hardened models and building blocks, including [Guardrail](/concepts/defense/guardrail) models tuned for Agent safety, hardened LLMs optimized for specific tasks, and pre-validated components that reduce months of security work to days. - Continuous improvement powered by reinforcement learning over production telemetry. Darwin learns from real-world failures, edge cases, and behavior drift to keep your agents resilient over time. + Continuous improvement powered by reinforcement learning over production telemetry. Darwin learns from real-world failures, edge cases, and behavior drift to keep your Agents resilient over time. diff --git a/owner-guide/getting-started/introduction.mdx b/owner-guide/getting-started/introduction.mdx index 637a03d..4009d93 100644 --- a/owner-guide/getting-started/introduction.mdx +++ b/owner-guide/getting-started/introduction.mdx @@ -127,11 +127,20 @@ First you will need to **register** an Agent, then **run** an [Evaluation](/conc - Coming Soon + Open **Agents** in the Console, select the Agent you want to protect, and click **Protect** in the Actions row. + + 1. Review the predefined **Input Guards** and **Output Guards**. + 2. Enable, disable, or extend the Guards based on the Agent's evaluation findings and policies. + 3. Test the Guard configuration with representative inputs. + 4. Click **Save**, then **Apply** when the configuration is ready. + + See [Configure Guardrails](/owner-guide/protect-in-production/configuring-guardrails) for the complete owner workflow. - - Coming Soon + + Return to the selected Agent and click **Monitor** in the Actions row. Review blocked inbound and outbound traffic, Guard errors, P99 latency, threat activity, and individual events. + + See [Monitor Dome Metrics](/owner-guide/protect-in-production/observability) for guidance on interpreting the dashboard. diff --git a/owner-guide/protect-in-production/observability.mdx b/owner-guide/protect-in-production/observability.mdx index fa54a87..b8e2d0d 100644 --- a/owner-guide/protect-in-production/observability.mdx +++ b/owner-guide/protect-in-production/observability.mdx @@ -1,176 +1,145 @@ --- -title: 'Observe Traces and Metrics' -description: 'Monitor Guardrail performance and agent traffic in real time.' +title: 'Monitor Dome Metrics' +description: 'Monitor Dome Guardrail performance and Agent traffic.' --- -The Observability Dashboard provides visibility into your protected agents, showing [Guardrail](/concepts/defense/guardrail) activity, latency metrics, and execution traces. +Dome Metrics provides visibility into a protected [Agent](/owner-guide/register-agents/registering-agents), including [Guardrail](/concepts/defense/guardrail) activity, blocked traffic, errors, latency, and execution details. Use these metrics to investigate threats, tune protection, and understand Dome's effect on production traffic. -## Accessing Telemetry +## Open Dome Metrics -1. Navigate to **Guardrails** in the sidebar -2. Find your agent in the Registered Agents table -3. Click the **Monitor** icon (chart) in the Monitor column +Monitor Dome from the Agent whose production traffic you want to review: -The telemetry dashboard opens for that agent. +1. Open **Agents** in the Console sidebar. +2. Select the Agent you want to monitor. +3. In the Actions row, click **Monitor**. +4. Review the Agent's **Dome Metrics**. -## Dashboard Layout + +[Dome](/owner-guide/protect-in-production/configuring-guardrails) must be configured for the Agent and sending telemetry before the dashboard can display production data. + -The dashboard has three main sections: +## Choose a Time Range -### 1. Guardrail Metrics -Summary cards at the top show aggregate statistics: +Use the time-range selector to review activity from the last **15m**, **30m**, **1h**, **2h**, **6h**, **1d**, **7d**, or **30d**. Summary metrics, charts, threat information, and events update for the selected range. -**Input Guardrails:** -- **Flagged Requests**: Number and percentage of requests that triggered input Guards -- **Errors**: Count of processing errors in input Guards + +Chart timestamps display in UTC. + -**Output Guardrails:** -- **Flagged Responses**: Number and percentage of responses that triggered output Guards -- **Errors**: Count of processing errors in output Guards +## Check Protection Status -### 2. Time Series Charts +The status banner provides a quick assessment of the selected time range and shows when the status was last updated. A **Clear** status means that Dome detected no threats during that interval. It does not guarantee that the Agent is free from every threat. -Four charts visualize Guardrail activity over time: +## Review Summary Metrics -| Chart | What It Shows | -|-------|---------------| -| **Input Flagged Requests Over Time** | Bar chart comparing flagged vs. total input requests | -| **Output Flagged Responses Over Time** | Bar chart comparing flagged vs. total output responses | -| **Input Latency Distribution** | Latency percentiles (P50, P90, P95, P99) for input Guards | -| **Output Latency Distribution** | Latency percentiles (P50, P90, P95, P99) for output Guards | - -### 3. Execution Traces & Logs - -Real-time monitoring of individual Guardrail executions: - -**Traces**: Detailed execution logs for each request, showing: -- Request ID -- Guards executed -- Detection results -- Timing information - -**Logs**: Streaming log output with search and level filtering. - -## Dashboard Controls - -### Layout Options - -Toggle between: -- **Grid**: Charts arranged in a grid layout -- **Row**: Charts stacked vertically +The summary cards show the Agent's protection activity for the selected time range: -### Chart Type +| Metric | What It Shows | +|--------|---------------| +| **Traffic** | Agent traffic recorded by Dome | +| **Inbound Blocked** | External requests stopped before reaching the Agent | +| **Outbound Blocked** | Agent responses stopped before reaching users because of policy violations | +| **Errors** | Guard failures recorded while processing traffic | +| **Latency P99** | The 99th-percentile latency overhead added by Dome | -Select how data points are displayed: -- **Line Connected**: Points connected with lines -- **Line Gaps**: Lines with gaps for missing data -- **Points Only**: Individual data points without connecting lines +**Latency P99** means that 99% of observed Guard operations completed within the displayed time. Track it over time to understand Dome's effect on the slowest requests. -### Time Range +## Read Traffic Charts -Filter data by time period: -- Last 24 Hours -- Last 7 Days -- Last 30 Days -- Custom range +The dashboard separates traffic by direction: -All times display in UTC. +| Chart | Traffic Direction | +|-------|-------------------| +| **Inbound** | External requests moving toward the Agent | +| **Outbound** | Agent responses moving toward users | -## Reading the Charts +Each chart compares blocked activity with total traffic. Red bars represent blocked requests or responses, while dark bars represent total traffic. -### Flagged Requests/Responses +An increase in blocked traffic can indicate: -The bar charts show two series: -- **Red bars**: Requests/responses flagged by Guardrails -- **Dark bars**: Total requests/responses +- Active threats or policy violations +- Guards that are too sensitive for normal traffic +- Changes in user behavior or Agent output -Hover over any bar to see exact counts and timestamps. +Compare the charts with **Threat Breakdown**, **Guard Performance**, and individual events before changing the Guard configuration. -A high flag rate may indicate: -- Active threats being blocked -- Overly sensitive Guard configuration (false positives) -- User behavior patterns that need attention +## Investigate Threats and Guard Activity -### Latency Distribution +Use the dashboard's investigation panels to understand what contributed to the summary metrics: -The latency charts show percentile lines: -- **P50 (Median)**: Half of requests complete faster than this -- **P90**: 90% of requests complete faster than this -- **P95**: 95% of requests complete faster than this -- **P99**: 99% of requests complete faster than this - -Monitor P99 latency to understand worst-case user experience. Spikes in P99 may indicate: -- Guard processing bottlenecks -- Resource contention -- Complex inputs requiring more analysis - -## Execution Traces - -The Traces panel shows recent Guardrail executions. Click any trace to view: +| Panel | How To Use It | +|-------|---------------| +| **Threat Breakdown** | Review the threat categories detected during the selected time range | +| **Guard Performance** | Identify which Guards are active and where protection may require tuning | +| **Events** | Inspect individual detection events for the selected filter | -- **Request ID**: Unique identifier for the request -- **Timestamp**: When the request was processed -- **Guards Executed**: Which Guards ran and in what order -- **Detection Results**: Whether each Guard flagged the content -- **Timing Breakdown**: How long each Guard took +Select an event to inspect its details. Use event-level information to determine whether a block represents an expected policy decision, an active threat, or a possible false positive. -Use traces to debug: -- Why specific requests were flagged -- Which Guards are slowest -- How Guards interact in your pipeline +## Open Logs and Traces -## Logs Panel +The **Logs & Traces** panel provides links to the detailed observability tools associated with the Agent: -The Logs panel shows streaming log output from your Guardrails. +- **Guard Execution Logs** provides operational output from Guard executions. +- **Trace Explorer** provides request-level traces across Guardrails, Guards, and Detectors. -### Filtering Logs +Follow a link to open the corresponding tool, or use its copy control to copy the URL. These logs and traces are not embedded in the Dome Metrics dashboard. -Filter by log level: -- **All Levels**: Show everything -- **Error**: Only errors -- **Warning**: Warnings and above -- **Info**: Informational and above -- **Debug**: All log levels +Use **Trace Explorer** to investigate: -Use the search box to find logs containing specific text. +- Why a request or response was blocked +- Which Guards and Detectors ran +- Whether each component flagged content or encountered an error +- Which Guards contributed the most execution time -### Real-time Updates +For instrumentation and trace details, see [Developer Observability](/developer-guide/protect/observability). To query raw telemetry from the command line, see the [Telemetry CLI](/developer-guide/cli/telemetry). -Toggle **Real-time Updates** to enable or disable automatic log streaming. Disable when you need to examine a specific log entry without the view scrolling. +## Interpret Metrics -## Interpreting Metrics +Healthy behavior depends on the Agent's purpose and traffic. Establish a normal baseline instead of relying on a universal block-rate threshold. -### Healthy Patterns +Healthy patterns generally include: -- **Low flag rate** (< 5%) with consistent traffic -- **Stable latency** without significant P99 spikes -- **Few errors** in Guardrail processing +- Traffic that follows expected usage patterns +- Blocked activity that remains consistent with the Agent's risk profile +- Stable P99 latency +- Few Guard errors -### Warning Signs +Investigate sustained changes: | Pattern | Possible Cause | Action | |---------|----------------|--------| -| High flag rate (> 20%) | Attack in progress or config too sensitive | Review flagged content, adjust thresholds | -| Increasing flag rate | Emerging threat or behavior change | Investigate new patterns | -| Latency spikes | Resource constraints or complex inputs | Review Guard performance, consider parallel execution | -| Error spikes | Guard failures or backend issues | Check logs for specific errors | +| Increasing inbound blocks | Attack activity, new user behavior, or sensitive input Guards | Review **Events**, **Threat Breakdown**, and traces | +| Increasing outbound blocks | Changes in Agent output or sensitive output Guards | Inspect blocked events and review the output Guard configuration | +| Error increase | Guard failures, deployment issues, or telemetry problems | Open **Guard Execution Logs** and verify the Dome deployment | +| P99 latency increase | Slow Guards, resource pressure, or unusually complex content | Review **Guard Performance** and compare Guard execution settings | +| Missing or unexpected traffic | No traffic in the selected interval or missing telemetry | Broaden the time range and verify deployment, Agent traffic, and telemetry | -## Exporting Data +Telemetry can support security investigations, compliance reviews, and performance analysis. Review trends regularly and compare changes with Guard configuration updates and Agent releases. -Use telemetry data for: -- Compliance reporting -- Security audits -- Performance analysis +## Empty States -Export options are available through the API. +Messages such as **No threats detected**, **No guard data available**, or **No events for the selected filter** can be expected when the selected interval contains no relevant activity. + +If the dashboard remains empty when you expect data: + +1. Select a broader time range. +2. Confirm that Dome is deployed for this Agent. +3. Confirm that the Agent received traffic during the selected interval. +4. Verify that Dome telemetry is reaching Vijil. ## Next Steps - + - Integrate Dome into your agent + Integrate Dome with the Agent - Adjust Guard configuration + Adjust the Agent's protection + + + Configure logs, traces, and metrics + + + Query raw telemetry