Da-2038 auditability phase1 synced

DOCKER.md

@@ -234,6 +234,14 @@ The detailed explanation for the environment variables can be found on the [GitH
 | `CB_MCP_OAUTH_MCP_BASE_URL`          | Public base URL of this server. When set, publishes RFC 9728 Protected Resource Metadata for PRM-aware clients                                            | None                                                           |
 | `CB_MCP_OAUTH_SCOPE_READ_LABEL`      | Override the OAuth scope label treated as 'read' access (advertised in PRM and matched against the token `scope`/`scp` claim). Use when your IdP can't emit the canonical form | `couchbase-mcp:read`                       |
 | `CB_MCP_OAUTH_SCOPE_WRITE_LABEL`     | Override the OAuth scope label treated as 'write' access; same semantics as the read label                                                                | `couchbase-mcp:write`                                          |
+| `CB_MCP_AUDIT_LOG_ENABLED`           | Enable audit logging — a separate sink from the `CB_MCP_LOG_*` operational logs, with a stable schema and its own retention (see [Auditing](#auditing))   | `false`                                                        |
+| `CB_MCP_AUDIT_LOG_SINKS`             | Comma-separated audit destinations: `console`, `file`, or both. `console` writes to **stderr** — never stdout, which carries the JSON-RPC protocol on stdio | `console`                                    |
+| `CB_MCP_AUDIT_LOG_FILE_PATH`         | Path to the audit log file. Required when `file` is in the sinks; the hostname and process id are inserted before the extension (`audit.log` → `audit.<host>.<pid>.log`), so containers sharing a volume never share a file. Mount a volume at this path to keep records after the container stops | None |
+| `CB_MCP_AUDIT_LOG_ROTATION_MAX_SIZE_MB` | Maximum size **in MB** the live audit file may reach before it rotates. `0` turns size-based rotation **off**                                          | `10` (10 MB)                                                   |
+| `CB_MCP_AUDIT_LOG_ROTATION_INTERVAL` | Age the live audit file may reach before it rotates, as `<value><unit>` where the unit is `d` (24 hours) or `w` (7 days) — e.g. `1d`, `30d`, `8w`. `0` turns interval-based rotation **off** | `1d`                         |
+| `CB_MCP_AUDIT_LOG_RETENTION_MAX_BACKUPS` | Rotated backups retained per audit file, excluding the live file; they are gzipped. Bounds one file, not the directory: each container start creates a new file with its own backups | `10`                   |
+| `CB_MCP_AUDIT_LOG_TOOL_ARGS`         | Record tool argument values. **Off by default** — there is no redaction in this release, so enabling it writes arguments verbatim, including full document bodies | `false`                                                |
+| `CB_MCP_AUDIT_LOG_DISABLED_EVENTS`   | Audit events to suppress: comma-separated numeric event ids (e.g. `61490,61491`), or a file with one id per line. Only filterable events can be suppressed | None                                            |
 | `EMBEDDING_PROVIDER`                 | Embedding provider for `run_vector_query` / `run_vector_search`: one of `couchbase`, `openai`, `cohere`, `voyage`, `bedrock`. Unset disables both tools' embedding step until configured. | None                                                           |
 | `EMBEDDING_MODEL`                    | Model name/ID for the configured embedding provider                                                                                                       | None                                                           |
 | `EMBEDDING_API_KEY`                  | API key for the configured provider. Not used by `bedrock` (uses the AWS credential chain / `EMBEDDING_AWS_*` instead)                                    | None                                                           |
@@ -420,6 +428,36 @@ The server logs to `stderr` by default. Logging is configured with the `CB_MCP_L
 
 For more details, see the [documentation](https://docs.couchbase.com/mcp-server/configuration/logging.html).
 
+### Auditing
+
+Audit logging records **who did what, and whether it was allowed** — a different question from the operational logs, which record what went wrong. It is off by default and enabled with the `CB_MCP_AUDIT_*` variables in the [Environment Variables](#environment-variables) table.
+
+Records are JSON Lines, one immutable record per line, in the same field vocabulary as Couchbase Server and Sync Gateway.
+
+- **`CB_MCP_AUDIT_LOG_ENABLED` / `CB_MCP_AUDIT_LOG_SINKS` / `CB_MCP_AUDIT_LOG_FILE_PATH`** — enabling auditing alone writes to the console (stderr). A container that keeps records needs `CB_MCP_AUDIT_LOG_SINKS=file` and a path. Selecting the file sink without a path leaves the server running, with the console sink if one was also selected, and an error in the log.
+- **Mount a volume at the audit path.** The audit file is written inside the container like any other file, so without a volume the records are destroyed when the container is removed — which defeats the purpose of keeping them. This matters more than it does for logs: the records are meant to outlive the process.
+- **One file per container.** The container's hostname and process id are inserted before the extension (`audit.log` → `audit.3f2a91c4b7de.1.log`). The hostname is what makes this safe: this image's `ENTRYPOINT` is exec form, so the server is **PID 1 in every container**, and a name built from the pid alone would give every container sharing a volume the same `audit.1.log` — interleaved lines and rotation races that silently destroy records. Docker assigns each container a distinct hostname by default. If you override `--hostname` to the same value on two containers **and** share one volume between them, give each its own subdirectory instead. Consolidating the files is the operator's job.
+- **Retention is per file, not per deployment.** `CB_MCP_AUDIT_LOG_RETENTION_MAX_BACKUPS` bounds the backups of *one* audit file. Each new container is a new hostname and so a new file with its own backups, and nothing prunes the files of containers that have exited. On a long-lived mounted volume, total usage grows with the number of container starts — size it accordingly, or sweep old `audit.*.log*` files yourself.
+- **Rotation and compression.** Size and age are independent triggers and either can be switched off with `0`: `CB_MCP_AUDIT_LOG_ROTATION_MAX_SIZE_MB` caps the live file's size, `CB_MCP_AUDIT_LOG_ROTATION_INTERVAL` caps its age, and whichever falls due first rotates it. Rotated files are named with the UTC rotation time and **gzipped** (`audit.3f2a91c4b7de.1.log.20261006T091422481503Z.gz`), which is what keeps a months-long retention window affordable on a mounted volume; the live file stays plain text so `docker exec ... tail -f` still works.
+- **Filtering** — `CB_MCP_AUDIT_LOG_DISABLED_EVENTS` turns read noise down, taking numeric event ids (e.g. `61490,61491`). Write and security events cannot be disabled; an attempt to do so is refused with a warning.
+- **`CB_MCP_AUDIT_LOG_TOOL_ARGS`** — off by default, and warns at startup when enabled. There is no redaction in this release, so arguments are written verbatim, including full document bodies.
+- **Fail-open** — if the audit file cannot be written, the server keeps serving and records are dropped and counted. `get_server_configuration_status` reports the live `written` / `dropped` / `write_errors` counters.
+
+Auditing applies to **both** servers. Each has its own Tier-2 event block, so an id names the server that emitted it: operational uses `0xF000` (61440), `operational-insights` `0x10000` (65536). The two share the core block `0xE000` (57344) for lifecycle, session, authorization and guardrail events.
+
+```bash
+docker run --rm \
+  -e CB_CONNECTION_STRING=couchbases://your-cluster \
+  -e CB_USERNAME=user -e CB_PASSWORD=password \
+  -e CB_MCP_AUDIT_LOG_ENABLED=true \
+  -e CB_MCP_AUDIT_LOG_SINKS=file \
+  -e CB_MCP_AUDIT_LOG_FILE_PATH=/audit/audit.log \
+  -e CB_MCP_AUDIT_LOG_ROTATION_INTERVAL=1d \
+  -e CB_MCP_AUDIT_LOG_RETENTION_MAX_BACKUPS=90 \
+  -v "$(pwd)/audit:/audit" \
+  couchbase/mcp-server-couchbase:latest
+```
+
 ### OAuth 2.1 Authorization
 
 When running with `CB_MCP_TRANSPORT=http`, the server can act as an **OAuth 2.1 resource server**: it validates incoming bearer JWTs against your identity provider's JWKS. It is provider-agnostic (any OAuth 2.1 / OIDC provider that publishes a JWKS — Auth0, Okta, Keycloak, AWS Cognito, Microsoft Entra, etc.) and does **not** issue tokens or manage users. OAuth settings are ignored on `stdio`.

README.md

@@ -328,6 +328,14 @@ The server can be configured using environment variables or command line argumen
 | `CB_MCP_OAUTH_MCP_BASE_URL` | `--oauth-mcp-base-url` | Public base URL of this server. When set, publishes RFC 9728 Protected Resource Metadata so PRM-aware clients can discover the IdP | None |
 | `CB_MCP_OAUTH_SCOPE_READ_LABEL` | `--oauth-scope-read-label` | Override the OAuth scope label treated as 'read' access (advertised in PRM and matched against the token's `scope`/`scp` claim). Use when your IdP can't emit the canonical form | `couchbase-mcp:read` |
 | `CB_MCP_OAUTH_SCOPE_WRITE_LABEL` | `--oauth-scope-write-label` | Override the OAuth scope label treated as 'write' access; same semantics as the read label | `couchbase-mcp:write` |
+| `CB_MCP_AUDIT_LOG_ENABLED` | `--audit-log-enabled` | Enable audit logging — a separate sink from the `CB_MCP_LOG_*` operational logs, with a stable schema and its own retention (see [Auditing](#auditing)) | `false` |
+| `CB_MCP_AUDIT_LOG_SINKS` | `--audit-log-sinks` | Comma-separated audit destinations: `console`, `file`, or both. `console` writes to **stderr** — never stdout, which carries the JSON-RPC protocol on stdio | `console` |
+| `CB_MCP_AUDIT_LOG_FILE_PATH` | `--audit-log-file-path` | Path to the audit log file. Required when `file` is in the sinks; the hostname and process id are inserted before the extension (`audit.log` → `audit.<host>.<pid>.log`) | None |
+| `CB_MCP_AUDIT_LOG_ROTATION_MAX_SIZE_MB` | `--audit-log-rotation-max-size-mb` | Maximum size **in MB** the live audit file may reach before it rotates. `0` turns size-based rotation **off** | `10` (10 MB) |
+| `CB_MCP_AUDIT_LOG_ROTATION_INTERVAL` | `--audit-log-rotation-interval` | Age the live audit file may reach before it rotates, as `<value><unit>` where the unit is `d` (24 hours) or `w` (7 days) — e.g. `1d`, `30d`, `8w`. `0` turns interval-based rotation **off** | `1d` |
+| `CB_MCP_AUDIT_LOG_RETENTION_MAX_BACKUPS` | `--audit-log-retention-max-backups` | Rotated backups retained **per audit file**, excluding the live file. Backups are gzipped. `0` keeps none. This bounds one file, not the directory — see [Auditing](#auditing) | `10` |
+| `CB_MCP_AUDIT_LOG_TOOL_ARGS` | `--audit-log-tool-args` | Record tool argument values in the audit log. **Off by default** — there is no redaction in this release, so enabling it writes arguments verbatim, including full document bodies | `false` |
+| `CB_MCP_AUDIT_LOG_DISABLED_EVENTS` | `--audit-log-disabled-events` | Audit events to suppress: comma-separated numeric event ids (e.g. `61490,61491`), or a file with one id per line. Only filterable events can be suppressed | None |
 | `EMBEDDING_PROVIDER` | `--embedding-provider` | Embedding provider for `run_vector_query` / `run_vector_search`: one of `couchbase`, `openai`, `cohere`, `voyage`, `bedrock`. Unset disables both tools' embedding step until configured — see [Embedding Provider Configuration](#embedding-provider-configuration) | None |
 | `EMBEDDING_MODEL` | `--embedding-model` | Model name/ID for the configured embedding provider | None |
 | `EMBEDDING_API_KEY` | `--embedding-api-key` | API key for the configured provider. Not used by `bedrock` (uses the AWS credential chain / `EMBEDDING_AWS_*` instead) | None |
@@ -542,6 +550,65 @@ uvx couchbase-mcp-server --log-level=debug --log-sinks=file \
 
 For more details, see the [documentation](https://docs.couchbase.com/mcp-server/configuration/logging.html).
 
+### Auditing
+
+Audit logging answers a different question from the operational logs. Logs tell you **what went wrong**; the audit log tells you **who did what, and whether it was allowed**. It exists because the MCP server is the only layer that can answer either question about an agent: every request reaches the cluster as the single `CB_USERNAME` the server connects with, so Couchbase Server's own audit log cannot tell two callers apart — and decisions such as a scope denial, a read-only block or a declined confirmation never reach the cluster at all.
+
+Auditing is **off by default**. `CB_MCP_AUDIT_LOG_ENABLED=true` turns it on, writing to the console (stderr) unless `CB_MCP_AUDIT_LOG_SINKS` says otherwise. A deployment that keeps records sets `CB_MCP_AUDIT_LOG_SINKS=file` and a path in `CB_MCP_AUDIT_LOG_FILE_PATH`; selecting the file sink without a path leaves the server running — with the console sink if one was also selected, otherwise with auditing off — and an error in the log.
+
+Records are **JSON Lines** — one immutable record per line — using the same field vocabulary as Couchbase Server and Sync Gateway (`id`, `name`, `description`, `timestamp`, `real_userid`, `server`, `cid`, `outcome`, `reason`), so an existing audit pipeline gets one format across the estate.
+
+```json
+{"id":61522,"name":"document write","description":"A document write operation was requested through a tool","timestamp":"2026-09-29T09:14:22.418Z","real_userid":{"domain":"oauth","user":"agent-svc-7"},"server":{"version":"1.0.2","host":"mcp-01"},"cid":"9f2c...","outcome":"success","tool_name":"upsert_document_by_id","required_scope":"write","ks":"travel-sample.inventory.airline","cb_userid":"mcp_service"}
+```
+
+- **Identity** — `real_userid` records the *caller*, in one of three domains: `oauth` (the bearer token's subject), `local` (the OS process owner, on stdio), or `anonymous` (HTTP with OAuth disabled). Enabling auditing on HTTP without OAuth warns at startup, because every record then resolves to `anonymous`.
+- **Correlation** — `cid` groups every record produced by one request, so an authorization denial, a guardrail refusal and the eventual tool result join without matching on timestamps. There is deliberately **no session identifier**: MCP is moving to a stateless model in which connection identity must not be read as session continuity. For SQL++, the same `cid` is sent as the query's `client_context_id`, which Couchbase Server records as `clientContextId` — an exact join between the two logs.
+- **Events** — a fixed catalogue on numbered ID blocks: server lifecycle and configuration, session initialize, token rejected, scope check denied, read-only write blocked, confirmation declined or skipped, and one read/write pair per tool category (cluster, schema, document, query, index, performance, search, config). Adding a tool needs no new ID and no change to a SIEM rule. The full catalogue ships as `descriptor.json` inside the package.
+- **Filtering** — `CB_MCP_AUDIT_LOG_DISABLED_EVENTS` turns *read* noise down, taking numeric event ids (the id is the wire contract; event names are prose and may be reworded, so they are not accepted). Write and security events are **not filterable**: an attempt to disable one is refused with a warning, so the record of who changed what cannot be switched off by configuration.
+- **Rotation** — two independent triggers, and either may be switched off with `0`. `CB_MCP_AUDIT_LOG_ROTATION_MAX_SIZE_MB` rotates the live file when it grows past a size; `CB_MCP_AUDIT_LOG_ROTATION_INTERVAL` rotates it when it grows older than an age (`1d`, `30d`, `1w`, `8w`). Whichever falls due first rotates the file, and with both off the file simply keeps growing. The interval clock is anchored to the live file, not to startup, so a server that was down over the weekend rolls the stale file away rather than appending to it.
+- **Retention and compression** — `CB_MCP_AUDIT_LOG_RETENTION_MAX_BACKUPS` keeps that many rotated files beside the live one; `0` keeps none, truncating the live file at each rotation. Backups are named with the UTC rotation time (`audit.<host>.<pid>.log.20261006T091422481503Z.gz`) and **gzipped**, which matters because audit retention is measured in months — JSON Lines compresses roughly ten to one. The live file is deliberately left uncompressed so `tail -f` and `grep` still work during an incident.
+- **Per-writer files** — the hostname and process id are inserted before the extension (`audit.log` → `audit.<host>.<pid>.log`), because a stdio deployment runs one server process per client and a shared rotating file would interleave and corrupt records. The hostname matters in containers: the published image's `ENTRYPOINT` is exec form, so the server is PID 1 in every container and a pid-only name would collide across containers sharing a volume.
+- **Retention is per file** — the backup count bounds the rotated copies of *one* audit file. Every process, and so every stdio session and every restart, writes a new file with its own backups, and nothing prunes the files of processes that have exited. Total disk use therefore grows with the number of server starts; size the volume accordingly or sweep old `audit.*.log*` files.
+- **Fail-open** — if the audit file cannot be written, the server keeps serving and the records are dropped and counted rather than turning an audit outage into a server outage. `get_server_configuration_status` reports the live `written` / `dropped` / `write_errors` counters, and `dropped` is the only way to discover that records were lost.
+- **Tool arguments** — recorded only when `CB_MCP_AUDIT_LOG_TOOL_ARGS` is enabled, which warns at startup. There is no redaction in this release, so arguments are written verbatim, including the full document bodies passed to document-write tools.
+
+Auditing applies to **both** servers. Each has its own Tier-2 event block, so an id names the server that emitted it: operational uses `0xF000` (61440), Operational Insights `0x10000` (65536). The two share the core block `0xE000` (57344) for lifecycle, session, authorization and guardrail events.
+
+```bash
+# Audit to the console (stderr) — no path needed
+uvx couchbase-mcp-server --audit-log-enabled=true
+
+# Audit to a file: daily rotation, 10 gzipped backups, both by default
+uvx couchbase-mcp-server --audit-log-enabled=true \
+  --audit-log-sinks=file --audit-log-file-path=/var/log/cb-mcp/audit.log
+
+# 90 days of daily files, with no size limit on any of them
+uvx couchbase-mcp-server --audit-log-enabled=true \
+  --audit-log-sinks=file --audit-log-file-path=/var/log/cb-mcp/audit.log \
+  --audit-log-rotation-max-size-mb=0 \
+  --audit-log-rotation-interval=1d \
+  --audit-log-retention-max-backups=90
+
+# ~100 MB of records, ~20 MB on disk once gzipped
+uvx couchbase-mcp-server --audit-log-enabled=true \
+  --audit-log-sinks=file --audit-log-file-path=/var/log/cb-mcp/audit.log \
+  --audit-log-rotation-max-size-mb=10 \
+  --audit-log-rotation-interval=0 \
+  --audit-log-retention-max-backups=9
+
+# One file, never rotated — keep everything
+uvx couchbase-mcp-server --audit-log-enabled=true \
+  --audit-log-sinks=file --audit-log-file-path=/var/log/cb-mcp/audit.log \
+  --audit-log-rotation-max-size-mb=0 --audit-log-rotation-interval=0 \
+  --audit-log-retention-max-backups=0
+
+# Turn down read noise, keeping every write and security event
+uvx couchbase-mcp-server --audit-log-enabled=true \
+  --audit-log-sinks=file --audit-log-file-path=/var/log/cb-mcp/audit.log \
+  --audit-log-disabled-events="61490,61491"   # document read, query read
+```
+
 ### Client Specific Configuration
 
 <details>