diff --git a/CHANGELOG.md b/CHANGELOG.md index e9a47f4..e25cff3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,16 +6,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ## [Unreleased] +## [0.12.0] - 2026-07-18 + ### Added +- **Command permissions** for `shell` / `ssh`: segment-aware allowlist, denylist, and confirm modes (`tools.*.permissions` in config). YAML baseline plus runtime overrides (SQLite), enforced **before** CommandGuard / user confirm. REST API under `/api/permissions/commands/*` (get/put overrides, dry-run validate with optional draft policy). See [docs/SECURITY.md](docs/SECURITY.md) +- **Alembic migrations** for SQLite: chat DB (sessions, tools, monitor, connectors, **canvas**) and **prompt_lab** DB. Auto-upgrade on web boot; Docker `agentforge-web` entrypoint runs `upgrade-all` before uvicorn/SAQ. Legacy DBs are stamped (no re-CREATE). Applied history in `schema_migrations` (`revision`, **filename**, `applied_at`). CLI: `python -m web.server.database.cli upgrade-all|upgrade|current|applied|history|revision`. See [docs/architecture.md](docs/architecture.md#sqlite-schema-alembic) - CI **Type Check (ty)** job: [Astral `ty`](https://docs.astral.sh/ty/) (Rust-based, pairs with Ruff) on `agentforge`, `chunking`, `sidecar`, `sandbox`, `tests` - `[tool.ty]` configuration in `pyproject.toml` with `app/` and `web/` excluded until diagnostics are cleared; `ty` added to the `dev` extra +- `alembic` dependency and console entry point `agentforge-db` ### Changed - **Dependencies simplified to production + dev only**: all runtime packages (framework, tools, service stack, SQLAlchemy drivers) live in `[project.dependencies]`; `[project.optional-dependencies]` now has only `dev` (ruff, ty, pytest). Removed granular extras (`bedrock`, `browser`, `service`, `all`, …) - CI runs on **pull requests only** (removed `push` to `master`) so merge does not duplicate the same lint/build/typecheck pass and added `concurrency` to cancel stale PR runs - Cleared all `ty` diagnostics in `agentforge/` (138 → 0): `chat()` overloads, tool registry typing, Playwright wait literals, config coercion helpers, and assorted narrowings +- `ChatDatabase.create_tables()` / Canvas / Prompt Lab schema setup use Alembic instead of ad-hoc `create_all` + `ALTER TABLE` blocks +- Shell `allowed_commands` / `blocked_patterns` still work as legacy keys under `tools.shell` when `permissions.*` lists are empty + +### Removed + +- Hand-written SQLite column migrations inside `web/server/database/manager.py` (replaced by Alembic revisions) ## [0.11.0] - 2026-06-28 diff --git a/Dockerfile.web b/Dockerfile.web index 68ee9be..e9cf349 100644 --- a/Dockerfile.web +++ b/Dockerfile.web @@ -42,6 +42,13 @@ COPY plugins/ plugins/ RUN mkdir -p /app/data /app/data/uploads +# Run Alembic migrations for chat + prompt_lab before starting the process. +# Workers reuse this image; their command is overridden in compose and still +# goes through this entrypoint so schema is ready before job handling. +COPY web/entrypoint.sh /entrypoint.sh +RUN chmod +x /entrypoint.sh + EXPOSE 8200 +ENTRYPOINT ["/entrypoint.sh"] CMD ["uvicorn", "web.server.app:app", "--host", "0.0.0.0", "--port", "8200"] diff --git a/README.md b/README.md index 358e00b..63eaaeb 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ These projects use AgentForge as their backend and don't run without it: - **Backends**: Ollama (local + cloud relay), AWS Bedrock, and any OpenAI-compatible API (DeepInfra, OpenRouter, ...). Selected per role via named profiles. Switch the whole stack with one `provider_override`. - **Agent loop**: think -> act -> observe with tool calling, error recovery, and optional web-search escalation. -- **Tools**: filesystem, shell, system info, Docker, Git, SSH, archives, network diagnostics, web search/fetch/render, media, code editing, macOS notifications, **Apple Reminders** (list/create/edit/complete/delete via `remindctl`), and more. +- **Tools**: filesystem, shell, system info, Docker, Git, SSH, archives, network diagnostics, web search/fetch/render, media, code editing, macOS notifications, **Apple Reminders** (list/create/edit/complete/delete via `remindctl`), and more. Shell/SSH commands can be gated by allowlist, denylist, or confirm policy (config + runtime API). - **RAG**: index OpenAPI/SQL schemas, source code, docs, and transcripts into Qdrant. Query with refinement, reranking, and dedup. - **Knowledge Database**: personal store for notes, references, documentation, attached documents, cheatsheets, and snippets. One-call ingest, semantic search, tag faceting, and smart updates (re-embeds only when content changes). Multi-collection routing (`X-Knowledge-Collection` header) separates the KB SPA and AgentForge Notes into independent Qdrant collections. Original attachment files (PDFs, etc.) are stored alongside extracted text and downloadable via the API. - **Connectors**: link external accounts as agent tools. Gmail, Drive, BigQuery, and YouTube through one Google OAuth client, plus GitLab and GitHub via personal access tokens. Multi-account, in-process, read-only by default. @@ -42,7 +42,7 @@ These projects use AgentForge as their backend and don't run without it: Operator guides live in [`docs/`](docs/README.md): -- [Stack architecture](docs/architecture.md): how the containers fit together: services, ports, worker localities, data stores, and request flow. **Start here.** +- [Stack architecture](docs/architecture.md): how the containers fit together: services, ports, worker localities, data stores, request flow, and **SQLite Alembic migrations**. **Start here.** - [HTTP API](docs/api.md): REST + the `/ws/chat` agent WebSocket, memory endpoints, the Knowledge Database (`/knowledge/*`), and the live OpenAPI spec. - [Modes](docs/modes.md): the `@mode` prefixes (built-in modes, custom agents, connectors) and when to use each. - [Tools](docs/tools.md): every built-in agent tool by category, plus locality and confirmation gates. @@ -51,7 +51,7 @@ Operator guides live in [`docs/`](docs/README.md): - [Connectors](docs/connectors.md): linking external accounts. The unified Google OAuth connector (Gmail, Drive, BigQuery, YouTube) and the GitLab and GitHub token connectors. - [Authoring tools and private overlays](docs/plugin-authoring.md): adding private tools, the `AGENTFORGE_TOOL_PLUGINS` seam, and the local overlay files. - [Instruction markdown](markdown/README.md): the `skills/` and `custom-agents/` markdown you edit to tune agents without touching Python. -- [Security](docs/SECURITY.md): the auth model, sidecar/internal tokens, interactive sudo, SSRF and read-only guards. +- [Security](docs/SECURITY.md): auth, sidecar/internal tokens, interactive sudo, SSRF and read-only guards, and **shell/SSH command permissions**. ## Run it locally @@ -160,10 +160,15 @@ ruff check . # lint ruff format --check . # formatting ty check # type check (phased scope; see pyproject.toml) pytest # tests + +python -m web.server.database.cli upgrade-all +python -m web.server.database.cli applied --database chat # revision + filename log ``` CI runs lint, format, type check, and build + smoke on pull requests (`.github/workflows/ci.yml`). +Schema changes belong in Alembic revision files under `web/server/database/migrations/` (chat/canvas) or `web/server/prompt_lab/database/migrations/` — see [architecture → SQLite schema](docs/architecture.md#sqlite-schema-alembic). + To exercise the framework directly without the Docker/web stack, see the [sandbox harness](sandbox/README.md): a short Python script driving `AIClient` / `ToolRegistry` / the agent loop against your Ollama. ## License diff --git a/agentforge/db_cli.py b/agentforge/db_cli.py new file mode 100644 index 0000000..d5c5aed --- /dev/null +++ b/agentforge/db_cli.py @@ -0,0 +1,19 @@ +"""Console entry point for chat-database migrations. + +Installed as ``agentforge-db`` when the package is installed editable +(``pip install -e .``). In package-mode Poetry envs you can also run:: + + python -m web.server.database.cli upgrade +""" + +from __future__ import annotations + + +def main() -> None: + from web.server.database.cli import main as cli_main + + raise SystemExit(cli_main()) + + +if __name__ == "__main__": + main() diff --git a/agentforge/tools/command_guard.py b/agentforge/tools/command_guard.py index 6f987f8..9a3a305 100644 --- a/agentforge/tools/command_guard.py +++ b/agentforge/tools/command_guard.py @@ -32,6 +32,9 @@ from chalkbox.logging.bridge import get_logger +from agentforge.client import AIClient +from agentforge.config import get_config + if TYPE_CHECKING: pass @@ -109,8 +112,6 @@ def _get_guard_config() -> dict: """Return guard config from config.yaml → tools.shell.guard.""" try: - from agentforge.config import get_config - cfg = get_config() return cfg._raw.get("tools", {}).get("shell", {}).get("guard", {}) except Exception: @@ -183,9 +184,6 @@ def _get_client(self): We use dataclasses.replace to clone with the guard's overrides. """ if self._client is None: - from agentforge.client import AIClient - from agentforge.config import get_config - shared = get_config().get_profile("fast") guard_profile = dataclasses.replace( shared, diff --git a/agentforge/tools/command_policy.py b/agentforge/tools/command_policy.py new file mode 100644 index 0000000..bfee3a9 --- /dev/null +++ b/agentforge/tools/command_policy.py @@ -0,0 +1,154 @@ +"""Segment-aware command permission evaluator for shell and ssh tools.""" + +from __future__ import annotations + +import logging +import os +import re +from dataclasses import dataclass +from typing import Literal + +from agentforge.config import get_config + +# Share shell parsing semantics with readonly_guard (segment splitting). +from agentforge.tools.readonly_guard import _split_segments + +logger = logging.getLogger(__name__) + +PolicyMode = Literal["confirm", "allowlist", "denylist"] +PolicyAction = Literal["allow", "deny", "confirm"] +ToolName = Literal["shell", "ssh"] + + +@dataclass(frozen=True) +class CommandPolicy: + mode: PolicyMode = "confirm" + allowed_commands: tuple[str, ...] = () + allowed_patterns: tuple[str, ...] = () + blocked_patterns: tuple[str, ...] = () + + +@dataclass(frozen=True) +class PolicyVerdict: + action: PolicyAction + reason: str = "" + source: str = "" + + +def _segment_base_command(segment: str) -> str: + parts = segment.strip().split() + idx = 0 + while idx < len(parts) and "=" in parts[idx] and not parts[idx].startswith("-"): + idx += 1 + while idx < len(parts) and parts[idx] in ("sudo", "command", "nice", "ionice", "time"): + idx += 1 + if idx >= len(parts): + return "" + return os.path.basename(parts[idx].rsplit("/", 1)[-1]) + + +def _pattern_matches(pattern: str, segment: str) -> bool: + try: + return bool(re.search(pattern, segment, re.IGNORECASE)) + except re.error: + logger.warning("Invalid regex pattern in command policy: %s", pattern) + return False + + +def _segment_passes_allowlist(segment: str, policy: CommandPolicy) -> bool: + base = _segment_base_command(segment) + if policy.allowed_commands and base in policy.allowed_commands: + return True + for pattern in policy.allowed_patterns: + if _pattern_matches(pattern, segment): + return True + return False + + +def _find_blocked_segment(command: str, policy: CommandPolicy) -> tuple[str, str] | None: + for segment in _split_segments(command): + for pattern in policy.blocked_patterns: + if _pattern_matches(pattern, segment): + return segment, pattern + return None + + +def evaluate(tool: ToolName, command: str, policy: CommandPolicy) -> PolicyVerdict: + if not command.strip(): + return PolicyVerdict(action="allow", reason="empty", source="noop") + + if policy.mode == "confirm": + blocked = _find_blocked_segment(command, policy) + if blocked is not None: + _segment, pattern = blocked + return PolicyVerdict( + action="deny", + reason=f"Command matches blocked pattern ({pattern})", + source="policy_blocked_pattern", + ) + return PolicyVerdict(action="confirm", reason="", source="policy_confirm") + + if policy.mode == "denylist": + blocked = _find_blocked_segment(command, policy) + if blocked is not None: + _segment, pattern = blocked + return PolicyVerdict( + action="deny", + reason=f"Command segment matches blocked pattern ({pattern})", + source="policy_denylist", + ) + return PolicyVerdict(action="allow", reason="", source="policy_denylist") + + # allowlist mode + for segment in _split_segments(command): + if not _segment_passes_allowlist(segment, policy): + base = _segment_base_command(segment) + return PolicyVerdict( + action="deny", + reason=f"Command '{base}' is not allowed", + source="policy_allowlist", + ) + return PolicyVerdict(action="allow", reason="", source="policy_allowlist") + + +def load_yaml_policy(tool: ToolName) -> CommandPolicy: + try: + cfg = get_config() + tool_cfg = cfg._raw.get("tools", {}).get(tool, {}) + perms = tool_cfg.get("permissions", {}) + + mode = perms.get("mode", "confirm") + allowed_commands = tuple(perms.get("allowed_commands") or ()) + allowed_patterns = tuple(perms.get("allowed_patterns") or ()) + blocked_patterns = tuple(perms.get("blocked_patterns") or ()) + + if not allowed_commands: + legacy_allowed = tool_cfg.get("allowed_commands") or [] + if legacy_allowed: + allowed_commands = tuple(legacy_allowed) + + if not blocked_patterns: + legacy_blocked = tool_cfg.get("blocked_patterns") or [] + if legacy_blocked: + blocked_patterns = tuple(legacy_blocked) + + return CommandPolicy( + mode=mode, + allowed_commands=allowed_commands, + allowed_patterns=allowed_patterns, + blocked_patterns=blocked_patterns, + ) + except Exception: + logger.warning("Failed to load YAML command policy for %s; using defaults", tool) + return CommandPolicy() + + +def merge_policies(base: CommandPolicy, override: CommandPolicy | None) -> CommandPolicy: + if override is None: + return base + return CommandPolicy( + mode=override.mode, + allowed_commands=override.allowed_commands or base.allowed_commands, + allowed_patterns=override.allowed_patterns or base.allowed_patterns, + blocked_patterns=override.blocked_patterns or base.blocked_patterns, + ) diff --git a/agentforge/tools/command_policy_store.py b/agentforge/tools/command_policy_store.py new file mode 100644 index 0000000..abf1464 --- /dev/null +++ b/agentforge/tools/command_policy_store.py @@ -0,0 +1,112 @@ +"""SQLite-backed runtime command policy overrides.""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import TYPE_CHECKING, Literal + +from agentforge.config import get_config +from agentforge.tools.command_policy import CommandPolicy, load_yaml_policy, merge_policies + +if TYPE_CHECKING: + from web.server.database import ChatDatabase + +ToolName = Literal["shell", "ssh"] + +_db: ChatDatabase | None = None +_web_unavailable: bool = False + + +def _resolve_db_path() -> Path: + env_path = os.environ.get("AGENTFORGE_CHAT_DB") + if env_path: + return Path(env_path).expanduser() + + cfg = get_config() + raw_path = cfg._raw.get("web", {}).get("database_path", "data/agentforge_chat.db") + return Path(raw_path).expanduser() + + +def _try_get_db() -> ChatDatabase | None: + """Return the chat DB, or ``None`` when the web package is not installed.""" + global _db, _web_unavailable + if _db is not None: + return _db + if _web_unavailable: + return None + # Lazy: core wheel does not ship ``web`` (see package hatch targets). + try: + from web.server.database import ChatDatabase + except ImportError: + _web_unavailable = True + return None + + db_path = _resolve_db_path() + _db = ChatDatabase(db_path) + _db.create_tables() + return _db + + +def _require_db() -> ChatDatabase: + db = _try_get_db() + if db is None: + raise RuntimeError( + "Runtime command-policy overrides require the AgentForge web stack " + "(module 'web' is not installed). YAML tools.*.permissions still apply." + ) + return db + + +def reset_db() -> None: + """Reset the module-level database singleton (for tests).""" + global _db, _web_unavailable + _db = None + _web_unavailable = False + + +def set_db(db: ChatDatabase) -> None: + """Inject a ChatDatabase instance (for tests).""" + global _db, _web_unavailable + _db = db + _web_unavailable = False + + +def _policy_to_dict(policy: CommandPolicy) -> dict: + return { + "mode": policy.mode, + "allowed_commands": list(policy.allowed_commands), + "allowed_patterns": list(policy.allowed_patterns), + "blocked_patterns": list(policy.blocked_patterns), + } + + +def _dict_to_policy(data: dict) -> CommandPolicy: + return CommandPolicy( + mode=data.get("mode", "confirm"), + allowed_commands=tuple(data.get("allowed_commands") or ()), + allowed_patterns=tuple(data.get("allowed_patterns") or ()), + blocked_patterns=tuple(data.get("blocked_patterns") or ()), + ) + + +def get_runtime_override(tool: ToolName) -> CommandPolicy | None: + db = _try_get_db() + if db is None: + return None + data = db.get_command_policy_override(tool) + if data is None: + return None + return _dict_to_policy(data) + + +def set_runtime_override(tool: ToolName, policy: CommandPolicy) -> None: + _require_db().upsert_command_policy_override(tool, _policy_to_dict(policy)) + + +def clear_runtime_override(tool: ToolName | None = None) -> int: + return _require_db().delete_command_policy_override(tool) + + +def get_effective_policy(tool: ToolName) -> CommandPolicy: + return merge_policies(load_yaml_policy(tool), get_runtime_override(tool)) diff --git a/agentforge/tools/registry.py b/agentforge/tools/registry.py index 06a8127..d7aee3c 100644 --- a/agentforge/tools/registry.py +++ b/agentforge/tools/registry.py @@ -11,18 +11,46 @@ from __future__ import annotations import asyncio +import concurrent.futures import inspect import typing from collections.abc import Callable +from dataclasses import dataclass from types import FunctionType -from typing import Any +from typing import Any, cast from chalkbox.logging.bridge import get_logger +from agentforge.config import get_config +from agentforge.tools.command_guard import get_guard +from agentforge.tools.command_policy import ToolName, evaluate +from agentforge.tools.command_policy_store import get_effective_policy +from agentforge.tools.routing import ( + _LEGACY_LOCALITY_MAP, + check_decorator_drift, + get_role_for_tool, + my_role, +) from agentforge.typing_utils import ToolCallable, callable_name logger = get_logger(__name__) +_saq_dispatch_tool: Callable[..., str] | None = None +try: + from web.server.queue.dispatch_compat import saq_dispatch_tool as _saq_dispatch_impl + + _saq_dispatch_tool = _saq_dispatch_impl +except ImportError: + pass + + +@dataclass(frozen=True) +class _CommandPolicyOutcome: + """Result of policy evaluation before CommandGuard.""" + + cancel_message: str | None = None + skip_guard: bool = False + # --------------------------------------------------------------------------- # @tool decorator @@ -367,6 +395,24 @@ def get_model_hints_condensed(self) -> str: # -- guard & confirmation ------------------------------------------------ + def _evaluate_command_policy(self, name: str, args: dict[str, Any]) -> _CommandPolicyOutcome: + """Evaluate YAML command policy for shell/ssh before CommandGuard.""" + if name not in ("shell", "ssh"): + return _CommandPolicyOutcome() + command = (args or {}).get("command") or "" + if not command.strip(): + return _CommandPolicyOutcome() + tool = cast(ToolName, name) + policy = get_effective_policy(tool) + verdict = evaluate(tool, command, policy) + if verdict.action == "deny": + return _CommandPolicyOutcome( + cancel_message=(f"Refused: command blocked by policy ({verdict.source}). {verdict.reason}"), + ) + if verdict.action == "allow": + return _CommandPolicyOutcome(skip_guard=True) + return _CommandPolicyOutcome() + def _classify_guard(self, func: Callable, args: dict[str, Any]) -> dict | None: """Run the CommandGuard for the shell + ssh tools (if applicable). @@ -388,8 +434,6 @@ def _classify_guard(self, func: Callable, args: dict[str, Any]) -> dict | None: return None try: - from .command_guard import get_guard - guard = get_guard() verdict = guard.classify(args["command"]) @@ -428,8 +472,6 @@ def _classify_guard(self, func: Callable, args: dict[str, Any]) -> dict | None: def _is_auto_sudo_enabled() -> bool: """Check if auto_sudo is enabled in config.yaml → tools.shell.auto_sudo.""" try: - from agentforge.config import get_config - cfg = get_config() return cfg._raw.get("tools", {}).get("shell", {}).get("auto_sudo", False) except Exception: @@ -446,8 +488,6 @@ def execute_with_role(self, name: str, args: dict[str, Any]) -> str: Use this wherever you'd otherwise call ``registry.execute("shell", ...)`` outside of the agent loop — parallel / discovery steps, etc. """ - from .routing import my_role - tool_role = self.get_role(name) worker_role = my_role() @@ -466,11 +506,7 @@ def execute_with_role(self, name: str, args: dict[str, Any]) -> str: tool_role, worker_role, ) - try: - from web.server.queue.dispatch_compat import saq_dispatch_tool - - return str(saq_dispatch_tool(name, args, tool_role)) - except ImportError: + if _saq_dispatch_tool is None: logger.error( "[Registry] dispatch_compat unavailable — running '%s' locally on '%s' worker despite tool_role='%s'", name, @@ -478,6 +514,7 @@ def execute_with_role(self, name: str, args: dict[str, Any]) -> str: tool_role, ) return str(self.execute(name, args)) + return str(_saq_dispatch_tool(name, args, tool_role)) def check_confirmation(self, name: str, args: dict[str, Any]) -> tuple[str | None, dict | None]: """Public entry point: run guard + confirm for a tool *without* executing it. @@ -495,7 +532,10 @@ def check_confirmation(self, name: str, args: dict[str, Any]) -> tuple[str | Non if func is None: return None, None coerced = self._coerce_args(func, args or {}) - guard_result = self._classify_guard(func, coerced) + policy_outcome = self._evaluate_command_policy(name, coerced) + if policy_outcome.cancel_message: + return policy_outcome.cancel_message, None + guard_result = None if policy_outcome.skip_guard else self._classify_guard(func, coerced) cancelled = self._check_confirm(func, coerced, guard_result) return cancelled, guard_result @@ -609,8 +649,6 @@ def get_role(self, name: str) -> str: covered by the catch-all ``"*"`` rule, but the fallback keeps the registry usable without a YAML present (e.g., in unit tests). """ - from .routing import _LEGACY_LOCALITY_MAP, get_role_for_tool - try: return get_role_for_tool(name) except Exception: @@ -625,8 +663,6 @@ def check_routing_drift(self) -> list[tuple[str, str, str]]: Logs a warning per mismatch and returns the drift list. Intended for startup observability — not a hard failure. """ - from .routing import check_decorator_drift - return check_decorator_drift(dict(self._tool_locality)) @staticmethod @@ -686,8 +722,12 @@ def execute( arg_pairs = ", ".join(f"'{k}': '{v}'" for k, v in args.items()) logger.debug("%s(%s)", name, arg_pairs) + policy_outcome = self._evaluate_command_policy(name, args) + if policy_outcome.cancel_message: + return policy_outcome.cancel_message + # Run guard BEFORE emitting tool_call so the UI can show the badge - guard_result = self._classify_guard(func, args) + guard_result = None if policy_outcome.skip_guard else self._classify_guard(func, args) if self._on_tool_call and not skip_events: try: @@ -716,8 +756,6 @@ def execute( loop = None if loop and loop.is_running(): - import concurrent.futures - with concurrent.futures.ThreadPoolExecutor() as pool: result = pool.submit(asyncio.run, func(**args)).result() else: @@ -753,8 +791,12 @@ async def aexecute(self, name: str, arguments: dict[str, Any] | None = None) -> arg_pairs = ", ".join(f"'{k}': '{v}'" for k, v in args.items()) logger.debug("%s(%s)", name, arg_pairs) + policy_outcome = self._evaluate_command_policy(name, args) + if policy_outcome.cancel_message: + return policy_outcome.cancel_message + # Run guard BEFORE emitting tool_call so the UI can show the badge - guard_result = self._classify_guard(func, args) + guard_result = None if policy_outcome.skip_guard else self._classify_guard(func, args) if self._on_tool_call: try: diff --git a/agentforge/tools/shell.py b/agentforge/tools/shell.py index e744d9c..5d901c5 100644 --- a/agentforge/tools/shell.py +++ b/agentforge/tools/shell.py @@ -33,8 +33,20 @@ from chalkbox.logging.bridge import get_logger +from agentforge.config import get_config +from agentforge.tools.command_policy import evaluate +from agentforge.tools.command_policy_store import get_effective_policy + from .registry import tool +OllamaClient: type | None = None +try: + from ollama import Client as _OllamaClientImpl + + OllamaClient = _OllamaClientImpl +except ImportError: + pass + if TYPE_CHECKING: from .registry import ToolRegistry @@ -121,33 +133,9 @@ def _invalidate_sudo_password(label: str = "localhost") -> None: # --------------------------------------------------------------------------- -def _get_allowed_commands() -> list[str]: - """Return the allowlist from config.yaml → tools.shell.allowed_commands.""" - try: - from agentforge.config import get_config - - cfg = get_config() - return cfg._raw.get("tools", {}).get("shell", {}).get("allowed_commands", []) - except Exception: - return [] - - -def _get_blocked_patterns() -> list[str]: - """Return blocked command patterns from config.yaml → tools.shell.blocked_patterns.""" - try: - from agentforge.config import get_config - - cfg = get_config() - return cfg._raw.get("tools", {}).get("shell", {}).get("blocked_patterns", []) - except Exception: - return [] - - def _get_default_timeout() -> int: """Return default timeout from config.yaml → tools.shell.timeout.""" try: - from agentforge.config import get_config - cfg = get_config() return int(cfg._raw.get("tools", {}).get("shell", {}).get("timeout", 600)) except Exception: @@ -162,8 +150,6 @@ def _auto_sudo_enabled() -> bool: *before* the rewrite never sees. Off by default; opt in explicitly. """ try: - from agentforge.config import get_config - cfg = get_config() return bool(cfg._raw.get("tools", {}).get("shell", {}).get("auto_sudo", False)) except Exception: @@ -443,11 +429,10 @@ def _fix_platform_compat(command: str) -> tuple[str, str]: if not _IS_MACOS or not _needs_platform_fix(command): return command, "" - try: - from ollama import Client as OllamaClient - - from agentforge.config import get_config + if OllamaClient is None: + return command, "" + try: cfg = get_config() fast = cfg.get_profile("fast") host = fast.host @@ -593,36 +578,14 @@ def _detect_agent_tool_misuse(command: str) -> str | None: def _validate_command(command: str) -> str | None: - """Validate a command against the allowlist and blocklist. + """Validate a command against the effective command policy. Returns an error string if denied, None if OK. """ - - # Check blocked patterns first - blocked = _get_blocked_patterns() - for pattern in blocked: - if re.search(pattern, command, re.IGNORECASE): - return ( - f"Error: Command matches a blocked pattern ({pattern}). " - f"This command is not permitted by the current configuration." - ) - - # Check allowlist (empty = allow all) - allowed = _get_allowed_commands() - if not allowed: - return None - - # Extract the base command (first word, ignoring env vars and paths) - cmd_parts = command.strip().split() - base_cmd = cmd_parts[0] if cmd_parts else "" - base_cmd = os.path.basename(base_cmd) - - if base_cmd not in allowed: - return ( - f"Error: Command '{base_cmd}' is not in the allowed commands list. " - f"Allowed: {', '.join(allowed)}. " - f"Add it to config.yaml → tools.shell.allowed_commands to permit it." - ) + policy = get_effective_policy("shell") + verdict = evaluate("shell", command, policy) + if verdict.action == "deny": + return f"Error: {verdict.reason}" return None diff --git a/agentforge/tools/ssh_tools.py b/agentforge/tools/ssh_tools.py index 986a8ea..502fd91 100644 --- a/agentforge/tools/ssh_tools.py +++ b/agentforge/tools/ssh_tools.py @@ -25,6 +25,10 @@ from chalkbox.logging.bridge import get_logger +from agentforge.config import get_config +from agentforge.tools.command_policy import evaluate +from agentforge.tools.command_policy_store import get_effective_policy + from .registry import tool if TYPE_CHECKING: @@ -79,8 +83,6 @@ def _run(cmd: list[str], timeout: int = 30, merge_stderr: bool = False) -> str: def _get_allowed_hosts() -> list[str]: """Return the allowed SSH hosts from config.""" try: - from ..config import get_config - cfg = get_config() hosts = cfg.get("tools.ssh.allowed_hosts") if isinstance(hosts, list): @@ -93,8 +95,6 @@ def _get_allowed_hosts() -> list[str]: def _get_connection_timeout() -> int: """Return the SSH connection timeout from config (default 10s).""" try: - from ..config import get_config - cfg = get_config() return int(cfg.get("tools.ssh.connection_timeout", 10)) except Exception: @@ -109,8 +109,6 @@ def _strict_host_key_checking() -> str: pre-provisioned known_hosts so a MITM on the first connection can't be trusted. """ try: - from ..config import get_config - cfg = get_config() return str(cfg._raw.get("tools", {}).get("ssh", {}).get("strict_host_key_checking", "accept-new")) except Exception: @@ -131,6 +129,18 @@ def _validate_host(host: str) -> str | None: return None +def _validate_ssh_command(command: str) -> str | None: + """Validate a remote command against the effective SSH command policy.""" + if not command.strip(): + return None + + policy = get_effective_policy("ssh") + verdict = evaluate("ssh", command, policy) + if verdict.action == "deny": + return f"Error: {verdict.reason}" + return None + + # --------------------------------------------------------------------------- # ssh — general-purpose remote execution # --------------------------------------------------------------------------- @@ -181,6 +191,10 @@ def ssh(host: str, command: str = "", timeout: int = 0) -> str: if err: return err + err = _validate_ssh_command(command) + if err: + return err + # Use config default if caller didn't specify a timeout if timeout <= 0: timeout = _get_connection_timeout() diff --git a/config.example.yaml b/config.example.yaml index 0aa4acd..7c85500 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -269,6 +269,16 @@ user_context: "user_context.md" tools: shell: timeout: 600 + # Command permission policy. YAML baseline for shell command strings. + # Runtime overrides (global, all sessions) can be set via the Web UI or + # /api/permissions/* and is merged at evaluation time. Checked segment-by-segment + # on compound commands (&&, |, ;) before CommandGuard / user confirm. + # Modes: confirm (default) | allowlist | denylist — see docs/SECURITY.md. + permissions: + mode: confirm + allowed_commands: [] # first-token allowlist; enforced when mode=allowlist + allowed_patterns: [] # regex; every segment must match ≥1 when mode=allowlist + blocked_patterns: [] # regex; any segment match → hard deny (all modes) # --- Privilege escalation (OFF by default; uncomment to opt in) --- # auto_sudo: false # When true, a file-mutating command targeting a root-owned path gets @@ -279,6 +289,13 @@ tools: # ones — there is no on-disk `sudo_password` anymore (removed). Reuse within # a session (no re-prompt within ~5 min) only happens in in-process dispatch; # the split-dispatch worker re-prompts on every sudo command. + ssh: + allowed_hosts: [] # host gate only — empty = any host; command policy is separate + permissions: + mode: confirm + allowed_commands: [] + allowed_patterns: [] + blocked_patterns: [] monitor: sites: {} # per-site extraction selectors (see docs); empty = generic extraction diff --git a/docs/README.md b/docs/README.md index fdc64ec..21aa0cc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,12 +2,12 @@ Guides for running and extending AgentForge. -- [architecture.md](architecture.md): how the Docker stack fits together: the services, ports, worker localities, data stores, and request flow. Start here. -- [api.md](api.md): the HTTP + WebSocket API: search/index endpoints, the Knowledge Database (`/knowledge/*`), the `/ws/chat` agent protocol, memory endpoints, and where the live OpenAPI spec lives. +- [architecture.md](architecture.md): how the Docker stack fits together: the services, ports, worker localities, data stores, request flow, and **Alembic chat-DB migrations**. Start here. +- [api.md](api.md): the HTTP + WebSocket API: search/index endpoints, the Knowledge Database (`/knowledge/*`), the `/ws/chat` agent protocol, memory endpoints, command permissions (`/api/permissions/*`), and where the live OpenAPI spec lives. - [api-examples.md](api-examples.md): runnable `curl` + `websocat` recipes, from a first prompt to processing the response, plus the in-prompt `@mode` / `#source` / `--flag` DSL. - [modes.md](modes.md): the `@mode` prefixes (built-in modes + custom agents + connectors), what each does, and when to use it. -- [tools.md](tools.md): every built-in agent tool, grouped by category, plus locality, confirmation gates, and how plugins add more. -- [SECURITY.md](SECURITY.md): the security controls (auth, sidecar/internal tokens, interactive sudo, SSRF and read-only guards). +- [tools.md](tools.md): every built-in agent tool, grouped by category, plus locality, confirmation gates, shell/SSH command policy, and how plugins add more. +- [SECURITY.md](SECURITY.md): auth, sidecar/internal tokens, interactive sudo, SSRF and read-only guards, and **shell/SSH command permissions**. - [model-catalog.md](model-catalog.md): comparing models across providers to find equivalents (`/api/model-catalog/*`), the per-provider catalog it draws on (`/api/catalog/*`), the `data/catalogs/*.json` files, the `catalog:*` Redis cache, and the `--with-catalog` deploy flow. - [chunking/README.md](../chunking/README.md): the chunking mappers (OpenAPI, SQL/tbls, live DB, code, CLI docs, Markdown), chunk layout, the index pipeline, the `/indexer/*` and `/search/*` endpoints, and dedup/drift QA. Served by the `agentforge-api` app on port `8100`. - [local-domains.md](local-domains.md): deploying the stack to a Mac/Linux box behind an existing Traefik proxy with custom domains, the `deploy.env` parameters, and the optional native (launchd) tool worker. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 07e6822..179db81 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -35,6 +35,67 @@ The sidecar (`agentforge-sidecar`, port `8300`) renders pages in a real browser, There is no on-disk `sudo_password`. When the `shell` tool needs sudo, the password is requested interactively over the `secret.request` / `secret.response` WebSocket pair (masked in the UI, masked `getpass` on the CLI). The value is memory-only — never persisted or logged. Auto-prepending `sudo` to root-targeted file mutations is opt-in via `tools.shell.auto_sudo` (default off), and even then it surfaces the same prompt — elevation is never silent. +## Command permissions (shell / SSH) + +AgentForge gates **shell and SSH command strings** through a hybrid policy layer before execution. Structured tools (`write_file`, `code_edit`, etc.,) are not covered here. + +**Bootstrap baseline** lives in `config.yaml` under `tools.shell.permissions` and `tools.ssh.permissions` (see `config.example.yaml`). **Runtime overrides** are stored in SQLite and apply globally to all chat sessions — they take effect on the next tool call without a restart. + +Policy is **segment-aware**: compound commands (`&&`, `|`, `;`) are split and each segment is evaluated independently. SSH `allowed_hosts` remains a separate host gate; command policy applies only to the remote `command` string. + +### Modes + +| Mode | Behavior | +|------|----------| +| `confirm` (default) | Hard deny when any segment matches `blocked_patterns`. Otherwise defer to **CommandGuard** (LLM + regex destructive-command detection) and the existing user-confirm flow. | +| `allowlist` | Hard deny when any segment fails `allowed_commands` (first token, after `sudo`/`nice`/…) or `allowed_patterns` (regex). **Skips CommandGuard** — allowed commands run without the destructive confirm dialog. | +| `denylist` | Hard deny when any segment matches `blocked_patterns`. Otherwise allow execution **without** CommandGuard confirm. | + +Denied commands never reach CommandGuard or the confirm dialog. In `allowlist` / `denylist` modes, policy errors fail closed (unknown → deny). CommandGuard itself still fails closed in `confirm` mode. + +### Configuration + +```yaml +tools: + shell: + permissions: + mode: confirm + allowed_commands: [] + allowed_patterns: [] + blocked_patterns: [] + ssh: + allowed_hosts: [] + permissions: + mode: confirm + allowed_commands: [] + allowed_patterns: [] + blocked_patterns: [] +``` + +**Legacy migration:** older configs may set `tools.shell.allowed_commands` or `tools.shell.blocked_patterns` at the tool root (as in `framework-config.example.yaml`). Those keys are still read when the matching `permissions.*` list is empty. Move them under `permissions.*` so the YAML baseline matches the REST API and Web UI. + +### REST API + +Mounted at `/api/permissions` (requires API-key auth when `security.api_keys` is set): + +| Method | Path | Purpose | +|--------|------|---------| +| `GET` | `/api/permissions/commands` | YAML baseline, runtime override, and merged **effective** policy per tool (`shell`, `ssh`) | +| `GET` | `/api/permissions/commands/overrides` | Runtime overrides only (`null` when unset) | +| `PUT` | `/api/permissions/commands/overrides` | Upsert overrides: `{shell?: {...}, ssh?: {...}}` | +| `DELETE` | `/api/permissions/commands/overrides` | Delete one override (`?tool=shell`) or all | +| `POST` | `/api/permissions/commands/validate` | Dry-run: `{tool, command, policy?}` → `{action, reason, source}` (`allow` \| `deny` \| `confirm`). Optional `policy` previews unsaved Web UI edits (merged with YAML as if saved). | + +Implementation: `web/server/permissions/api.py`, evaluation: `agentforge/tools/command_policy.py`. + +### Web UI + +Open **Command Permissions** from the chat input menu (alongside Profiles / Connectors). The modal edits runtime overrides per tool (mode, allow/deny lists), shows YAML vs effective policy, and includes a **Test command** validator that evaluates the **current form** (including unsaved edits). **Save** persists to SQLite; **Reset** clears overrides and restores the YAML baseline. + +### Relationship to CommandGuard + +Command policy runs **first** in the tool registry (`agentforge/tools/registry.py`). CommandGuard (`tools.shell.guard` in `framework-config.yaml`) applies only when policy returns `confirm`. Use `confirm` mode for the full LLM safety net; use `allowlist` or `denylist` when you want static rules without per-command confirmation. + ## Other Stuff - **SQL** (`@sql`): `readonly` connections are enforced at the transaction level (Postgres `SET TRANSACTION READ ONLY`; MySQL session hook + a never-commit backstop), not just by convention. diff --git a/docs/api.md b/docs/api.md index 664affd..fa2be70 100644 --- a/docs/api.md +++ b/docs/api.md @@ -413,6 +413,7 @@ Grouped by subsystem. See the live `/docs` for full request/response schemas. | Scheduler | `/api/scheduler/*` | Cron-style recurring agent jobs (APScheduler). | | Monitor | `/api/monitor/*` | Website-change monitors + checks. | | Connectors | `/api/connectors/*` | Google (Gmail/Drive/BigQuery/YouTube) OAuth + GitLab token connections (see [connectors.md](connectors.md)). | +| Permissions | `/api/permissions/commands/*` | Shell/SSH command policy: YAML baseline + SQLite runtime overrides, dry-run validate (see [SECURITY.md](SECURITY.md#command-permissions-shell--ssh)). | | Canvas | `/api/canvas/*` | Per-session pinned-items workspace. | | Configs | `/api/configs*` | Read-only view of whitelisted YAML config files. | | Services | `/api/services*` | Container/service health dashboard + log tail/stream. | diff --git a/docs/architecture.md b/docs/architecture.md index a4ee9d7..914eb72 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -102,6 +102,43 @@ These ship with `agentforge-web` but are independent of core chat/RAG. Each has Botty honors `analysis_interval` (process every Nth completed run), `max_frequency_seconds` (minimum gap between nudges), and `dismissal_cooldown_seconds` (quiet period after a dismiss). Model roles and the Qdrant `insights` collection are configured under `botty:` but not yet used by the engine. +## SQLite schema (Alembic) + +`agentforge-web` SQLite databases are versioned with **[Alembic](https://alembic.sqlalchemy.org/)**. + +| Database | File | Migrations | Marker table (legacy stamp) | +| -------- | ---- | ---------- | --------------------------- | +| **chat** (+ canvas) | `web.database_path` (e.g. `data/agentforge_chat.db`) | `web/server/database/migrations/` | `chat_sessions` | +| **prompt_lab** | `data/prompt_lab.db` | `web/server/prompt_lab/database/migrations/` | `prompt_lab_runs` | + +Canvas shares the **chat** file; `canvas_items` is revision `002_canvas_items`. + +| Concern | How it works | +| -------- | ------------ | +| **Docker / deploy** | `Dockerfile.web` `ENTRYPOINT` runs `python -m web.server.database.cli upgrade-all` before uvicorn / SAQ. | +| **App boot** | `create_tables()` also calls `upgrade` (idempotent safety net). | +| **Empty DB** | Baseline revisions create all tables. | +| **Existing pre-Alembic DB** | Marker table present + no `alembic_version` → **stamp** head (no re-CREATE). | +| **Applied history** | Table `schema_migrations` stores each applied `revision`, **filename**, and `applied_at` (Laravel-style log). Alembic’s `alembic_version` still holds the current head. | +| **New schema changes** | New files under `…/migrations/versions/` only — never ad-hoc `ALTER` in managers. | + +```bash +# All DBs (also the Docker entrypoint command) +python -m web.server.database.cli upgrade-all + +# Single DB +python -m web.server.database.cli upgrade --database chat +python -m web.server.database.cli upgrade --database prompt_lab + +# After editing models +python -m web.server.database.cli revision -m "add foo" --database chat --autogenerate + +# Inspect +python -m web.server.database.cli current --database chat +python -m web.server.database.cli applied --database chat # filename log +python -m web.server.database.cli history --database chat +``` + ## Configuration Two files at the repo root, both gitignored (copy from the `*.example.yaml`): diff --git a/docs/tools.md b/docs/tools.md index f3fa56e..8b29b32 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -44,7 +44,7 @@ The exact set exposed on any one run depends on the mode and the active profile: | ------- | --------------------------------------------------------- | | `shell` | Run a shell command locally (guarded by a command policy) | -Commands run argv-only (no `shell=True`). The destructive-command guard fails **closed** (treats anything it can't classify as destructive and pauses for confirmation). Sudo is never read from disk. If a command needs it, you're prompted interactively (WebSocket+API). Auto-elevating root-owned actions can be opt-in to via `tools.shell.auto_sudo`. See [SECURITY.md](SECURITY.md). +Commands run argv-only (no `shell=True`). **Policy first:** `tools.shell.permissions` (YAML) plus optional runtime overrides (`/api/permissions/*`) gate allowlist / denylist / confirm **before** CommandGuard. Then the destructive-command guard fails **closed** (unknown → treat as destructive and pause for confirmation). Sudo is never read from disk; if a command needs it, you're prompted interactively (WebSocket+API). Auto-elevating root-owned actions is opt-in via `tools.shell.auto_sudo`. See [SECURITY.md](SECURITY.md#command-permissions-shell--ssh). ## Docker @@ -81,6 +81,8 @@ Commands run argv-only (no `shell=True`). The destructive-command guard fails ** | `rsync` | Sync files or directories remotely | | `ssh_keygen` | Generate or inspect SSH keys | +Hosts must be on `tools.ssh.allowed_hosts`. Remote **command strings** use the same permission modes as shell (`tools.ssh.permissions` + runtime overrides); host allowlisting is separate from command policy. See [SECURITY.md](SECURITY.md#command-permissions-shell--ssh). + ## Network and diagnostics | Tool | Description | diff --git a/framework-config.example.yaml b/framework-config.example.yaml index d127ba0..e18093f 100644 --- a/framework-config.example.yaml +++ b/framework-config.example.yaml @@ -86,8 +86,14 @@ tools: max_tail_lines: 200 shell: timeout: 120 - allowed_commands: [] # empty = allow all; e.g., ["git", "npm", "python"] - blocked_patterns: [] # regex patterns to block, e.g., ["rm\\s+-rf\\s+/"] + permissions: + mode: confirm # confirm | allowlist | denylist + allowed_commands: [] # e.g., ["git", "npm", "python"] — used when mode=allowlist + allowed_patterns: [] # regex; every segment must match ≥1 when mode=allowlist + blocked_patterns: [] # e.g., ["rm\\s+-rf\\s+/"] — hard deny on any segment match + # Legacy: tools.shell.allowed_commands / tools.shell.blocked_patterns are still + # read when the matching permissions.* list is empty. Migrate to permissions.* + # above so YAML baseline aligns with /api/permissions and the Web UI. auto_sudo: false guard: enabled: true # LLM-based destructive-command detection @@ -117,8 +123,13 @@ tools: optimize_quality: 82 strip_metadata: true ssh: - allowed_hosts: [] # only these hosts may be reached by the ssh tool + allowed_hosts: [] # host gate only — command policy applies to the remote command string connection_timeout: 10 + permissions: + mode: confirm + allowed_commands: [] + allowed_patterns: [] + blocked_patterns: [] ignored_dirs: - .venv - node_modules diff --git a/poetry.lock b/poetry.lock index e78a375..f491d3b 100644 --- a/poetry.lock +++ b/poetry.lock @@ -170,6 +170,26 @@ files = [ frozenlist = ">=1.1.0" typing-extensions = {version = ">=4.2", markers = "python_version < \"3.13\""} +[[package]] +name = "alembic" +version = "1.18.5" +description = "A database migration tool for SQLAlchemy." +optional = false +python-versions = ">=3.10" +groups = ["main"] +files = [ + {file = "alembic-1.18.5-py3-none-any.whl", hash = "sha256:06d8ba9d04558022f5395e9317de03d270f3dced49cee01f89fe7a13c26f14bc"}, + {file = "alembic-1.18.5.tar.gz", hash = "sha256:1554982221dd17e9a749b53902407578eb305e453f71999e8c7f0a48389fff8e"}, +] + +[package.dependencies] +Mako = "*" +SQLAlchemy = ">=1.4.23" +typing-extensions = ">=4.12" + +[package.extras] +tz = ["tzdata"] + [[package]] name = "annotated-doc" version = "0.0.4" @@ -1806,6 +1826,26 @@ html-clean = ["lxml_html_clean"] html5 = ["html5lib"] htmlsoup = ["BeautifulSoup4"] +[[package]] +name = "mako" +version = "1.3.12" +description = "A super-fast templating language that borrows the best ideas from the existing templating languages." +optional = false +python-versions = ">=3.8" +groups = ["main"] +files = [ + {file = "mako-1.3.12-py3-none-any.whl", hash = "sha256:8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9"}, + {file = "mako-1.3.12.tar.gz", hash = "sha256:9f778e93289bd410bb35daadeb4fc66d95a746f0b75777b942088b7fd7af550a"}, +] + +[package.dependencies] +MarkupSafe = ">=0.9.2" + +[package.extras] +babel = ["Babel"] +lingua = ["lingua"] +testing = ["pytest"] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -1830,6 +1870,105 @@ profiling = ["gprof2dot"] rtd = ["ipykernel", "jupyter_sphinx", "mdit-py-plugins (>=0.5.0)", "myst-parser", "pyyaml", "sphinx", "sphinx-book-theme (>=1.0,<2.0)", "sphinx-copybutton", "sphinx-design"] testing = ["coverage", "pytest", "pytest-cov", "pytest-regressions", "pytest-timeout", "requests"] +[[package]] +name = "markupsafe" +version = "3.0.3" +description = "Safely add untrusted strings to HTML/XML markup." +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "markupsafe-3.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2f981d352f04553a7171b8e44369f2af4055f888dfb147d55e42d29e29e74559"}, + {file = "markupsafe-3.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e1c1493fb6e50ab01d20a22826e57520f1284df32f2d8601fdd90b6304601419"}, + {file = "markupsafe-3.0.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ba88449deb3de88bd40044603fafffb7bc2b055d626a330323a9ed736661695"}, + {file = "markupsafe-3.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f42d0984e947b8adf7dd6dde396e720934d12c506ce84eea8476409563607591"}, + {file = "markupsafe-3.0.3-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c0c0b3ade1c0b13b936d7970b1d37a57acde9199dc2aecc4c336773e1d86049c"}, + {file = "markupsafe-3.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f"}, + {file = "markupsafe-3.0.3-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:d2ee202e79d8ed691ceebae8e0486bd9a2cd4794cec4824e1c99b6f5009502f6"}, + {file = "markupsafe-3.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:177b5253b2834fe3678cb4a5f0059808258584c559193998be2601324fdeafb1"}, + {file = "markupsafe-3.0.3-cp310-cp310-win32.whl", hash = "sha256:2a15a08b17dd94c53a1da0438822d70ebcd13f8c3a95abe3a9ef9f11a94830aa"}, + {file = "markupsafe-3.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:c4ffb7ebf07cfe8931028e3e4c85f0357459a3f9f9490886198848f4fa002ec8"}, + {file = "markupsafe-3.0.3-cp310-cp310-win_arm64.whl", hash = "sha256:e2103a929dfa2fcaf9bb4e7c091983a49c9ac3b19c9061b6d5427dd7d14d81a1"}, + {file = "markupsafe-3.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad"}, + {file = "markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a"}, + {file = "markupsafe-3.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50"}, + {file = "markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf"}, + {file = "markupsafe-3.0.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f"}, + {file = "markupsafe-3.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a"}, + {file = "markupsafe-3.0.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115"}, + {file = "markupsafe-3.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a"}, + {file = "markupsafe-3.0.3-cp311-cp311-win32.whl", hash = "sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19"}, + {file = "markupsafe-3.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01"}, + {file = "markupsafe-3.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c"}, + {file = "markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e"}, + {file = "markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce"}, + {file = "markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d"}, + {file = "markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d"}, + {file = "markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a"}, + {file = "markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b"}, + {file = "markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f"}, + {file = "markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b"}, + {file = "markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d"}, + {file = "markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c"}, + {file = "markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f"}, + {file = "markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795"}, + {file = "markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219"}, + {file = "markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6"}, + {file = "markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676"}, + {file = "markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9"}, + {file = "markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1"}, + {file = "markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc"}, + {file = "markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12"}, + {file = "markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed"}, + {file = "markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5"}, + {file = "markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485"}, + {file = "markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73"}, + {file = "markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37"}, + {file = "markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19"}, + {file = "markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025"}, + {file = "markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6"}, + {file = "markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f"}, + {file = "markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb"}, + {file = "markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009"}, + {file = "markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354"}, + {file = "markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218"}, + {file = "markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287"}, + {file = "markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe"}, + {file = "markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026"}, + {file = "markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737"}, + {file = "markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97"}, + {file = "markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d"}, + {file = "markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda"}, + {file = "markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf"}, + {file = "markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe"}, + {file = "markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9"}, + {file = "markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581"}, + {file = "markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4"}, + {file = "markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab"}, + {file = "markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175"}, + {file = "markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634"}, + {file = "markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50"}, + {file = "markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e"}, + {file = "markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5"}, + {file = "markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523"}, + {file = "markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc"}, + {file = "markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d"}, + {file = "markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9"}, + {file = "markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa"}, + {file = "markupsafe-3.0.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:15d939a21d546304880945ca1ecb8a039db6b4dc49b2c5a400387cdae6a62e26"}, + {file = "markupsafe-3.0.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:f71a396b3bf33ecaa1626c255855702aca4d3d9fea5e051b41ac59a9c1c41edc"}, + {file = "markupsafe-3.0.3-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f4b68347f8c5eab4a13419215bdfd7f8c9b19f2b25520968adfad23eb0ce60c"}, + {file = "markupsafe-3.0.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8fc20152abba6b83724d7ff268c249fa196d8259ff481f3b1476383f8f24e42"}, + {file = "markupsafe-3.0.3-cp39-cp39-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:949b8d66bc381ee8b007cd945914c721d9aba8e27f71959d750a46f7c282b20b"}, + {file = "markupsafe-3.0.3-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:3537e01efc9d4dccdf77221fb1cb3b8e1a38d5428920e0657ce299b20324d758"}, + {file = "markupsafe-3.0.3-cp39-cp39-musllinux_1_2_riscv64.whl", hash = "sha256:591ae9f2a647529ca990bc681daebdd52c8791ff06c2bfa05b65163e28102ef2"}, + {file = "markupsafe-3.0.3-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:a320721ab5a1aba0a233739394eb907f8c8da5c98c9181d1161e77a0c8e36f2d"}, + {file = "markupsafe-3.0.3-cp39-cp39-win32.whl", hash = "sha256:df2449253ef108a379b8b5d6b43f4b1a8e81a061d6537becd5582fba5f9196d7"}, + {file = "markupsafe-3.0.3-cp39-cp39-win_amd64.whl", hash = "sha256:7c3fb7d25180895632e5d3148dbdc29ea38ccb7fd210aa27acbd1201a1902c6e"}, + {file = "markupsafe-3.0.3-cp39-cp39-win_arm64.whl", hash = "sha256:38664109c14ffc9e7437e86b4dceb442b0096dfe3541d7864d9cbe1da4cf36c8"}, + {file = "markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698"}, +] + [[package]] name = "maxminddb" version = "3.1.1" @@ -4096,4 +4235,4 @@ dev = ["pytest", "pytest-asyncio", "ruff", "ty"] [metadata] lock-version = "2.1" python-versions = ">=3.12,<4.0" -content-hash = "8d6c7224aaa52f10e55ce3b70ddbaf0c6e6bed62b58fa5352fcae50f9584a68d" +content-hash = "1c1803b81c603fb53c515d94fc4cde8e2537f0678ae5280dc967495c50602c24" diff --git a/pyproject.toml b/pyproject.toml index 8352d63..e754f08 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "agent-forge" -version = "0.11.0" +version = "0.12.0" description = "AgentForge — a composable, multi-backend agent + pipeline framework with a pluggable tool system." readme = "README.md" requires-python = ">=3.12,<4.0" @@ -17,23 +17,23 @@ classifiers = [ # (FastAPI RAG API, web/server workers, tools, connectors). No optional extras. dependencies = [ "aiohttp>=3.14.0", - "apscheduler>=3.10.0,<4", - "beautifulsoup4>=4.12.0", - "boto3>=1.35.0", + "apscheduler>=3.10.4", + "beautifulsoup4>=4.12.3", + "boto3>=1.35.99", "chalkbox>=2.3.5", - "cryptography>=42.0", - "curl_cffi>=0.7.0", - "deepdiff>=7.0.0", + "cryptography>=42.0.8", + "curl_cffi>=0.7.4", + "deepdiff>=7.0.1", "detect-secrets>=1.5.0", "docker>=7.1.0", - "fastapi>=0.115.0", - "google-auth>=2.0.0", - "httpx>=0.27.0", - "lxml>=5.0.0", - "maxminddb>=2.6.0", - "numpy>=1.26.0", - "ollama>=0.4.0", - "pdfplumber>=0.10.0", + "fastapi>=0.115.14", + "google-auth>=2.0.2", + "httpx>=0.27.2", + "lxml>=6.1.1", + "maxminddb>=3.1.1", + "numpy>=1.26.4", + "ollama>=0.4.9", + "pdfplumber>=0.10.4", "playwright>=1.40.0", "playwright-stealth>=2.0.0", "psycopg2-binary>=2.9.0", @@ -47,16 +47,23 @@ dependencies = [ "redis[hiredis]>=5.0.0", "saq[hiredis]>=0.12.0", "slack-bolt>=1.18.0", + "alembic>=1.18.5,<2.0.0", "sqlalchemy>=2.0.0", "timezonefinder>=6.5.0", "uvicorn[standard]>=0.49.0", ] [project.optional-dependencies] -dev = ["ruff>=0.8.0", "pytest>=8.0.0", "pytest-asyncio>=0.24.0", "ty>=0.0.55,<0.1"] +dev = [ + "ruff>=0.8.0", + "pytest>=8.0.0", + "pytest-asyncio>=0.24.0", + "ty>=0.0.55,<0.1", +] [project.scripts] agentforge = "agentforge.cli:main" +agentforge-db = "agentforge.db_cli:main" agentforge-chunk-openapi = "chunking.openapi.cli:main" agentforge-chunk-sql = "chunking.sql.cli:main" agentforge-chunk-code = "chunking.code.cli:main" @@ -109,7 +116,15 @@ ignore = ["E501", "N806", "E741", "N818"] # Otherwise ruff auto-detects first-party from the environment and classifies # `agentforge` as third-party in CI (its lint job doesn't install the package), # demanding a group-separating blank line the local run never asks for. -known-first-party = ["agentforge", "app", "web", "sidecar", "sandbox", "plugins", "chunking"] +known-first-party = [ + "agentforge", + "app", + "web", + "sidecar", + "sandbox", + "plugins", + "chunking", +] [tool.ruff.lint.per-file-ignores] # Intentional late imports: FastAPI apps register auth/routes after the app diff --git a/tests/test_command_policy.py b/tests/test_command_policy.py new file mode 100644 index 0000000..ee23469 --- /dev/null +++ b/tests/test_command_policy.py @@ -0,0 +1,63 @@ +from agentforge.tools.command_policy import ( + CommandPolicy, + evaluate, + merge_policies, +) + + +def test_allowlist_denies_unknown_command(): + policy = CommandPolicy(mode="allowlist", allowed_commands=("git", "ls")) + v = evaluate("shell", "npm install", policy) + assert v.action == "deny" + assert "not allowed" in v.reason.lower() + + +def test_allowlist_allows_chained_if_each_segment_ok(): + policy = CommandPolicy(mode="allowlist", allowed_commands=("git", "ls")) + v = evaluate("shell", "git status && ls -la", policy) + assert v.action == "allow" + + +def test_allowlist_denies_chained_with_writer(): + policy = CommandPolicy(mode="allowlist", allowed_commands=("git", "ls")) + v = evaluate("shell", "git status && npm install", policy) + assert v.action == "deny" + + +def test_denylist_blocks_pattern_any_segment(): + policy = CommandPolicy(mode="denylist", blocked_patterns=(r"git\s+push",)) + v = evaluate("shell", "git status && git push origin main", policy) + assert v.action == "deny" + assert v.source == "policy_denylist" + + +def test_confirm_mode_blocked_pattern_denies_without_confirm(): + policy = CommandPolicy( + mode="confirm", + blocked_patterns=(r"rm\s+-rf",), + ) + v = evaluate("shell", "rm -rf /tmp/x", policy) + assert v.action == "deny" + + +def test_confirm_mode_safe_command_defers_to_guard(): + policy = CommandPolicy(mode="confirm") + v = evaluate("shell", "git status", policy) + assert v.action == "confirm" + + +def test_allowed_pattern_matches_segment(): + policy = CommandPolicy( + mode="allowlist", + allowed_patterns=(r"^git\s+(status|log|diff)\b",), + ) + assert evaluate("shell", "git status", policy).action == "allow" + assert evaluate("shell", "git push", policy).action == "deny" + + +def test_merge_policies_override_wins(): + base = CommandPolicy(mode="confirm", blocked_patterns=(r"foo",)) + override = CommandPolicy(mode="allowlist", allowed_commands=("ls",)) + merged = merge_policies(base, override) + assert merged.mode == "allowlist" + assert merged.allowed_commands == ("ls",) diff --git a/tests/test_command_policy_integration.py b/tests/test_command_policy_integration.py new file mode 100644 index 0000000..54c2fc4 --- /dev/null +++ b/tests/test_command_policy_integration.py @@ -0,0 +1,107 @@ +from unittest.mock import patch + +from agentforge.tools.command_policy import CommandPolicy, PolicyVerdict +from agentforge.tools.registry import ToolRegistry +from agentforge.tools.ssh_tools import ssh + + +def test_registry_denies_before_confirm(monkeypatch): + reg = ToolRegistry() + prompts = [] + reg.set_confirm_handler(lambda p: prompts.append(p) or True) + + policy = CommandPolicy(mode="allowlist", allowed_commands=("ls",)) + + def fake_shell(command: str) -> str: + return "ok" + + reg.register(fake_shell, name="shell") + + with ( + patch("agentforge.tools.registry.get_effective_policy", return_value=policy), + patch( + "agentforge.tools.registry.evaluate", + return_value=PolicyVerdict(action="deny", reason="blocked", source="policy_allowlist"), + ), + ): + result = reg.execute("shell", {"command": "npm install"}) + + assert "blocked" in result.lower() or "not allowed" in result.lower() or "policy" in result.lower() + assert prompts == [] # confirm must NOT fire + + +def test_registry_allow_skips_command_guard(): + reg = ToolRegistry() + prompts = [] + reg.set_confirm_handler(lambda p: prompts.append(p) or True) + + policy = CommandPolicy(mode="allowlist", allowed_commands=("ls",)) + + def fake_shell(command: str) -> str: + return "ok" + + reg.register(fake_shell, name="shell") + + with ( + patch("agentforge.tools.registry.get_effective_policy", return_value=policy), + patch( + "agentforge.tools.registry.evaluate", + return_value=PolicyVerdict(action="allow", reason="", source="policy_allowlist"), + ), + patch.object(reg, "_classify_guard") as mock_guard, + ): + result = reg.execute("shell", {"command": "ls -la"}) + + assert result == "ok" + mock_guard.assert_not_called() + assert prompts == [] + + +def test_registry_confirm_runs_command_guard(): + reg = ToolRegistry() + prompts = [] + reg.set_confirm_handler(lambda p: prompts.append(p) or True) + + policy = CommandPolicy(mode="confirm") + + def fake_shell(command: str) -> str: + return "ok" + + reg.register(fake_shell, name="shell") + + guard_result = { + "source": "fast-path", + "destructive": False, + "sudo_only": False, + "auto_confirmed": False, + "verdict": "safe", + } + + with ( + patch("agentforge.tools.registry.get_effective_policy", return_value=policy), + patch( + "agentforge.tools.registry.evaluate", + return_value=PolicyVerdict(action="confirm", reason="", source="policy_confirm"), + ), + patch.object(reg, "_classify_guard", return_value=guard_result) as mock_guard, + ): + result = reg.execute("shell", {"command": "git status"}) + + assert result == "ok" + mock_guard.assert_called_once() + + +def test_ssh_policy_denied(): + policy = CommandPolicy(mode="allowlist", allowed_commands=("ls",)) + + with ( + patch("agentforge.tools.ssh_tools._validate_host", return_value=None), + patch( + "agentforge.tools.ssh_tools.get_effective_policy", + return_value=policy, + ), + ): + result = ssh("myserver", "npm install") + + assert "Error:" in result + assert "not allowed" in result.lower() or "npm" in result.lower() diff --git a/tests/test_command_policy_store.py b/tests/test_command_policy_store.py new file mode 100644 index 0000000..017ba26 --- /dev/null +++ b/tests/test_command_policy_store.py @@ -0,0 +1,87 @@ +import pytest + +from agentforge.config import reset_config +from agentforge.tools.command_policy import CommandPolicy +from agentforge.tools.command_policy_store import ( + clear_runtime_override, + get_effective_policy, + get_runtime_override, + reset_db, + set_db, + set_runtime_override, +) +from web.server.database import ChatDatabase + + +@pytest.fixture +def tmp_db(tmp_path, monkeypatch): + db_path = tmp_path / "test.db" + db = ChatDatabase(db_path) + db.create_tables() + reset_db() + set_db(db) + monkeypatch.setenv("AGENTFORGE_CHAT_DB", str(db_path)) + yield db_path + clear_runtime_override(None) + reset_db() + reset_config() + + +def test_runtime_override_overrides_yaml_mode(tmp_db, monkeypatch): + set_runtime_override("shell", CommandPolicy(mode="allowlist", allowed_commands=("ls",))) + p = get_effective_policy("shell") + assert p.mode == "allowlist" + assert p.allowed_commands == ("ls",) + clear_runtime_override("shell") + + +def test_get_runtime_override_none_when_empty(tmp_db): + assert get_runtime_override("shell") is None + + +def test_set_and_get_runtime_override(tmp_db): + policy = CommandPolicy( + mode="denylist", + blocked_patterns=(r"rm\s+-rf",), + ) + set_runtime_override("ssh", policy) + loaded = get_runtime_override("ssh") + assert loaded == policy + + +def test_clear_runtime_override_single_tool(tmp_db): + set_runtime_override("shell", CommandPolicy(mode="allowlist", allowed_commands=("ls",))) + set_runtime_override("ssh", CommandPolicy(mode="confirm")) + clear_runtime_override("shell") + assert get_runtime_override("shell") is None + assert get_runtime_override("ssh") is not None + clear_runtime_override("ssh") + + +def test_clear_runtime_override_all(tmp_db): + set_runtime_override("shell", CommandPolicy(mode="allowlist", allowed_commands=("ls",))) + set_runtime_override("ssh", CommandPolicy(mode="confirm")) + clear_runtime_override(None) + assert get_runtime_override("shell") is None + assert get_runtime_override("ssh") is None + + +def test_effective_policy_merges_partial_override(tmp_db, monkeypatch): + yaml_policy = CommandPolicy( + mode="confirm", + allowed_commands=("git", "ls"), + blocked_patterns=(r"sudo",), + ) + monkeypatch.setattr( + "agentforge.tools.command_policy_store.load_yaml_policy", + lambda _tool: yaml_policy, + ) + set_runtime_override( + "shell", + CommandPolicy(mode="allowlist", allowed_commands=(), blocked_patterns=()), + ) + effective = get_effective_policy("shell") + assert effective.mode == "allowlist" + assert effective.allowed_commands == ("git", "ls") + assert effective.blocked_patterns == (r"sudo",) + clear_runtime_override("shell") diff --git a/tests/test_load_yaml_policy.py b/tests/test_load_yaml_policy.py new file mode 100644 index 0000000..70f2782 --- /dev/null +++ b/tests/test_load_yaml_policy.py @@ -0,0 +1,106 @@ +from __future__ import annotations + +import pytest + +from agentforge.tools.command_policy import CommandPolicy, load_yaml_policy + + +class _FakeConfig: + def __init__(self, raw: dict) -> None: + self._raw = raw + + +@pytest.fixture +def fake_config(monkeypatch): + holder: dict = {"raw": {"tools": {}}} + + def _get_config(): + return _FakeConfig(holder["raw"]) + + # Patch the name bound in command_policy (top-level import). + monkeypatch.setattr("agentforge.tools.command_policy.get_config", _get_config) + return holder + + +def test_load_permissions_block(fake_config): + fake_config["raw"] = { + "tools": { + "shell": { + "permissions": { + "mode": "allowlist", + "allowed_commands": ["git", "ls"], + "allowed_patterns": [r"^npm\s+"], + "blocked_patterns": [r"rm\s+-rf"], + } + } + } + } + policy = load_yaml_policy("shell") + assert policy.mode == "allowlist" + assert policy.allowed_commands == ("git", "ls") + assert policy.allowed_patterns == (r"^npm\s+",) + assert policy.blocked_patterns == (r"rm\s+-rf",) + + +def test_legacy_fallback_when_permissions_empty(fake_config): + fake_config["raw"] = { + "tools": { + "shell": { + "allowed_commands": ["npm", "node"], + "blocked_patterns": [r"sudo\s+"], + "permissions": { + "mode": "confirm", + "allowed_commands": [], + "blocked_patterns": [], + }, + } + } + } + policy = load_yaml_policy("shell") + assert policy.mode == "confirm" + assert policy.allowed_commands == ("npm", "node") + assert policy.blocked_patterns == (r"sudo\s+",) + + +def test_legacy_only_without_permissions_key(fake_config): + fake_config["raw"] = { + "tools": { + "ssh": { + "allowed_commands": ["git"], + "blocked_patterns": [r"git\s+push"], + } + } + } + policy = load_yaml_policy("ssh") + assert policy.mode == "confirm" + assert policy.allowed_commands == ("git",) + assert policy.blocked_patterns == (r"git\s+push",) + + +def test_permissions_override_legacy_lists(fake_config): + fake_config["raw"] = { + "tools": { + "shell": { + "allowed_commands": ["legacy-cmd"], + "blocked_patterns": [r"legacy-block"], + "permissions": { + "mode": "denylist", + "allowed_commands": ["git"], + "blocked_patterns": [r"rm\s+"], + }, + } + } + } + policy = load_yaml_policy("shell") + assert policy.mode == "denylist" + assert policy.allowed_commands == ("git",) + assert policy.blocked_patterns == (r"rm\s+",) + + +def test_config_error_returns_defaults(monkeypatch): + def _boom(): + raise RuntimeError("no config") + + monkeypatch.setattr("agentforge.tools.command_policy.get_config", _boom) + policy = load_yaml_policy("shell") + assert policy == CommandPolicy() diff --git a/tests/test_migrations.py b/tests/test_migrations.py new file mode 100644 index 0000000..3abe466 --- /dev/null +++ b/tests/test_migrations.py @@ -0,0 +1,140 @@ +"""Tests for Alembic multi-database migrations + schema_migrations log.""" + +from __future__ import annotations + +from pathlib import Path + +from sqlalchemy import create_engine, inspect, text +from sqlalchemy.pool import NullPool + +from web.server.database.migrate import ( + current, + history, + list_applied, + upgrade, + upgrade_all, +) +from web.server.database.models import Base + +CHAT_HEAD = "002_canvas_items" +PROMPT_LAB_HEAD = "001_pl_baseline" + + +def test_upgrade_empty_chat_db_creates_schema(tmp_path: Path): + db = tmp_path / "empty.db" + upgrade(db, database="chat") + assert current(db, database="chat") == CHAT_HEAD + + engine = create_engine(f"sqlite:///{db}", poolclass=NullPool) + try: + tables = set(inspect(engine).get_table_names()) + finally: + engine.dispose() + + assert "chat_sessions" in tables + assert "command_policy_overrides" in tables + assert "canvas_items" in tables + assert "alembic_version" in tables + assert "schema_migrations" in tables + + +def test_schema_migrations_records_filenames(tmp_path: Path): + db = tmp_path / "logged.db" + upgrade(db, database="chat") + rows = list_applied(db, database="chat") + filenames = {r["filename"] for r in rows} + revisions = {r["revision"] for r in rows} + assert "001_baseline" in revisions + assert "002_canvas_items" in revisions + assert any("001_baseline" in f for f in filenames) + assert any("002_canvas_items" in f or "canvas" in f for f in filenames) + + +def test_upgrade_is_idempotent(tmp_path: Path): + db = tmp_path / "idem.db" + upgrade(db, database="chat") + upgrade(db, database="chat") + assert current(db, database="chat") == CHAT_HEAD + assert len(list_applied(db, database="chat")) == 2 + + +def test_legacy_db_is_stamped_not_recreated(tmp_path: Path): + db = tmp_path / "legacy.db" + engine = create_engine(f"sqlite:///{db}", poolclass=NullPool) + try: + Base.metadata.create_all(bind=engine) + with engine.begin() as conn: + conn.execute( + text( + "INSERT INTO chat_sessions (id, title, source, message_count, " + "prompt_tokens, completion_tokens, total_tokens, created_at, updated_at) " + "VALUES ('s1', 'Legacy', 'web', 0, 0, 0, 0, datetime('now'), datetime('now'))" + ) + ) + finally: + engine.dispose() + + assert current(db, database="chat") is None + upgrade(db, database="chat") + assert current(db, database="chat") == CHAT_HEAD + + engine = create_engine(f"sqlite:///{db}", poolclass=NullPool) + try: + with engine.connect() as conn: + row = conn.execute(text("SELECT title FROM chat_sessions WHERE id='s1'")).fetchone() + assert row is not None + assert row[0] == "Legacy" + finally: + engine.dispose() + + # Stamp path still populates schema_migrations + assert len(list_applied(db, database="chat")) >= 1 + + +def test_prompt_lab_baseline(tmp_path: Path): + db = tmp_path / "pl.db" + upgrade(db, database="prompt_lab") + assert current(db, database="prompt_lab") == PROMPT_LAB_HEAD + engine = create_engine(f"sqlite:///{db}", poolclass=NullPool) + try: + tables = set(inspect(engine).get_table_names()) + finally: + engine.dispose() + assert "prompt_lab_runs" in tables + assert "prompt_lab_results" in tables + rows = list_applied(db, database="prompt_lab") + assert any(r["revision"] == PROMPT_LAB_HEAD for r in rows) + + +def test_upgrade_all(tmp_path: Path, monkeypatch): + chat = tmp_path / "chat.db" + pl = tmp_path / "pl.db" + results = upgrade_all(chat_db=chat, prompt_lab_db=pl) + assert results["chat"] == CHAT_HEAD + assert results["prompt_lab"] == PROMPT_LAB_HEAD + + +def test_history_includes_chat_revisions(): + revs = history(database="chat") + assert "001_baseline" in revs + assert "002_canvas_items" in revs + + +def test_chat_database_create_tables_uses_alembic(tmp_path: Path): + from web.server.database.manager import ChatDatabase + + db_path = tmp_path / "via_manager.db" + ChatDatabase(db_path).create_tables() + assert current(db_path, database="chat") == CHAT_HEAD + + +def test_canvas_create_tables_uses_chat_alembic(tmp_path: Path): + from web.server.canvas.database import CanvasDatabase + + db_path = tmp_path / "canvas.db" + CanvasDatabase(db_path).create_tables() + engine = create_engine(f"sqlite:///{db_path}", poolclass=NullPool) + try: + assert "canvas_items" in inspect(engine).get_table_names() + finally: + engine.dispose() diff --git a/tests/test_permissions_api.py b/tests/test_permissions_api.py new file mode 100644 index 0000000..97b2e2d --- /dev/null +++ b/tests/test_permissions_api.py @@ -0,0 +1,149 @@ +from __future__ import annotations + +import pytest +from fastapi.testclient import TestClient + +from agentforge.config import reset_config +from agentforge.tools.command_policy_store import clear_runtime_override, reset_db, set_db +from web.server.app import app +from web.server.database import ChatDatabase + + +@pytest.fixture +def client(tmp_path, monkeypatch): + db_path = tmp_path / "test_permissions_api.db" + db = ChatDatabase(db_path) + db.create_tables() + reset_db() + set_db(db) + monkeypatch.setenv("AGENTFORGE_CHAT_DB", str(db_path)) + with TestClient(app) as test_client: + yield test_client + clear_runtime_override(None) + reset_db() + reset_config() + + +def test_get_effective_policy(client): + r = client.get("/api/permissions/commands") + assert r.status_code == 200 + data = r.json() + assert "shell" in data + assert "ssh" in data + assert "yaml" in data["shell"] + assert "override" in data["shell"] + assert "effective" in data["shell"] + assert data["shell"]["override"] is None + + +def test_validate_command(client): + r = client.post( + "/api/permissions/commands/validate", + json={"tool": "shell", "command": "git status"}, + ) + assert r.status_code == 200 + assert r.json()["action"] in ("allow", "deny", "confirm") + + +def test_get_overrides_empty(client): + r = client.get("/api/permissions/commands/overrides") + assert r.status_code == 200 + data = r.json() + assert data["shell"] is None + assert data["ssh"] is None + + +def test_put_and_get_overrides(client): + payload = { + "shell": { + "mode": "allowlist", + "allowed_commands": ["git", "ls"], + "allowed_patterns": [], + "blocked_patterns": [], + } + } + r = client.put("/api/permissions/commands/overrides", json=payload) + assert r.status_code == 200 + assert r.json() == {"ok": True} + + r = client.get("/api/permissions/commands/overrides") + assert r.status_code == 200 + assert r.json()["shell"]["mode"] == "allowlist" + assert r.json()["shell"]["allowed_commands"] == ["git", "ls"] + assert r.json()["ssh"] is None + + +def test_delete_single_override(client): + client.put( + "/api/permissions/commands/overrides", + json={"shell": {"mode": "denylist", "blocked_patterns": [r"rm\s+-rf"]}}, + ) + client.put( + "/api/permissions/commands/overrides", + json={"ssh": {"mode": "confirm"}}, + ) + + r = client.delete("/api/permissions/commands/overrides?tool=shell") + assert r.status_code == 200 + assert r.json()["deleted"] == 1 + + overrides = client.get("/api/permissions/commands/overrides").json() + assert overrides["shell"] is None + assert overrides["ssh"] is not None + + +def test_delete_all_overrides(client): + client.put( + "/api/permissions/commands/overrides", + json={ + "shell": {"mode": "allowlist", "allowed_commands": ["ls"]}, + "ssh": {"mode": "confirm"}, + }, + ) + + r = client.delete("/api/permissions/commands/overrides") + assert r.status_code == 200 + assert r.json()["deleted"] == 2 + + overrides = client.get("/api/permissions/commands/overrides").json() + assert overrides["shell"] is None + assert overrides["ssh"] is None + + +def test_validate_draft_policy_without_persisting(client): + """Draft policy in validate body previews unsaved Web UI edits.""" + r = client.post( + "/api/permissions/commands/validate", + json={ + "tool": "shell", + "command": "npm install", + "policy": { + "mode": "allowlist", + "allowed_commands": ["ls"], + "allowed_patterns": [], + "blocked_patterns": [], + }, + }, + ) + assert r.status_code == 200 + body = r.json() + assert body["action"] == "deny" + assert "not allowed" in body["reason"].lower() + + # No override persisted — effective policy unchanged. + assert client.get("/api/permissions/commands/overrides").json()["shell"] is None + + +def test_validate_denied_under_allowlist_override(client): + client.put( + "/api/permissions/commands/overrides", + json={"shell": {"mode": "allowlist", "allowed_commands": ["ls"]}}, + ) + r = client.post( + "/api/permissions/commands/validate", + json={"tool": "shell", "command": "npm install"}, + ) + assert r.status_code == 200 + body = r.json() + assert body["action"] == "deny" + assert "not allowed" in body["reason"].lower() diff --git a/tests/test_reminders_tools.py b/tests/test_reminders_tools.py index ba82bf3..7653a11 100644 --- a/tests/test_reminders_tools.py +++ b/tests/test_reminders_tools.py @@ -3,7 +3,7 @@ from __future__ import annotations import json -from datetime import datetime +from datetime import date, datetime, timedelta from unittest.mock import patch import agentforge.tools.reminders_tools as rt @@ -52,6 +52,10 @@ def test_reminders_add_builds_remindctl_args(monkeypatch): monkeypatch.setattr(rt, "_is_macos", lambda: True) monkeypatch.setattr(rt, "_has_remindctl", lambda: True) + # Absolute ISO dates must be on/after today (_validate_due_date); use a + # far-future day so the test does not expire as the calendar advances. + due = (date.today() + timedelta(days=30)).isoformat() + with patch.object( rt, "_run_remindctl", @@ -60,7 +64,7 @@ def test_reminders_add_builds_remindctl_args(monkeypatch): result = rt.reminders_add( "Buy milk", list_name="Groceries", - due_date="2026-06-30", + due_date=due, notes="2%", priority="high", ) @@ -73,7 +77,7 @@ def test_reminders_add_builds_remindctl_args(monkeypatch): "--list", "Groceries", "--due", - "2026-06-30", + due, "--notes", "2%", "--priority", diff --git a/web/entrypoint.sh b/web/entrypoint.sh new file mode 100644 index 0000000..d1b51ae --- /dev/null +++ b/web/entrypoint.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# agentforge-web entrypoint: apply SQLite migrations, then exec the container command. +# +# Migrations also run inside FastAPI lifespan (create_tables → upgrade). Running +# them here makes deploy logs explicit and fails the container before uvicorn +# if a migration cannot be applied. +set -e + +echo "[agentforge-web] Running database migrations (chat + prompt_lab)…" +python -m web.server.database.cli upgrade-all +echo "[agentforge-web] Migrations complete." + +exec "$@" diff --git a/web/server/app.py b/web/server/app.py index 4d85b4e..ac87687 100644 --- a/web/server/app.py +++ b/web/server/app.py @@ -54,6 +54,7 @@ from .database import ChatDatabase from .model_catalog.api import router as model_catalog_router from .monitor_service import init_monitor, shutdown_monitor +from .permissions.api import router as permissions_api_router from .prompt_lab.database.manager import PromptLabDatabase from .scheduler_service import init_scheduler, shutdown_scheduler from .services.api import router as services_api_router @@ -342,6 +343,9 @@ async def _log_requests(request: Request, call_next): # Connectors — external service integrations (Gmail, Drive, etc.) app.include_router(connectors_api_router) +# Command permissions: YAML baseline + SQLite runtime overrides +app.include_router(permissions_api_router) + # Botty — Session Awareness Layer (WebSocket); gated by botty.enabled in config.yaml if af_settings.botty.enabled: app.include_router(botty_router) diff --git a/web/server/canvas/database.py b/web/server/canvas/database.py index 10fa360..1b3686f 100644 --- a/web/server/canvas/database.py +++ b/web/server/canvas/database.py @@ -18,6 +18,8 @@ from sqlalchemy.orm import sessionmaker from sqlalchemy.pool import NullPool +from web.server.database.migrate import upgrade as alembic_upgrade + logger = logging.getLogger(__name__) @@ -45,30 +47,9 @@ def __init__(self, db_path: str | Path) -> None: logger.info("CanvasDatabase initialised at %s", self.db_path) def create_tables(self) -> None: - """Create canvas_items table and index if they don't exist (idempotent).""" - with self.engine.connect() as conn: - conn.execute( - text(""" - CREATE TABLE IF NOT EXISTS canvas_items ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - session_id TEXT NOT NULL, - type TEXT NOT NULL, - content TEXT NOT NULL, - label TEXT, - footnote_num INTEGER NOT NULL, - created_at TEXT NOT NULL, - UNIQUE(session_id, type, content) - ) - """) - ) - conn.execute( - text(""" - CREATE INDEX IF NOT EXISTS canvas_items_session - ON canvas_items(session_id, footnote_num) - """) - ) - conn.commit() - logger.info("Canvas tables ready") + """Apply chat-DB Alembic migrations (includes ``canvas_items``).""" + alembic_upgrade(self.db_path, database="chat") + logger.info("Canvas schema ready via chat Alembic at %s", self.db_path) def _row_to_dict(self, row) -> dict: return { diff --git a/web/server/canvas/models.py b/web/server/canvas/models.py new file mode 100644 index 0000000..e7886c2 --- /dev/null +++ b/web/server/canvas/models.py @@ -0,0 +1,28 @@ +"""Canvas items — ORM model on the shared chat SQLite database. + +Tables live in the same file as chat sessions (see ``CanvasDatabase``). +Schema is managed by the **chat** Alembic tree under +``web/server/database/migrations/``. +""" + +from __future__ import annotations + +from sqlalchemy import Column, Index, Integer, String, Text, UniqueConstraint + +from web.server.database.models import Base + + +class CanvasItem(Base): + __tablename__ = "canvas_items" + __table_args__ = ( + UniqueConstraint("session_id", "type", "content", name="uq_canvas_session_type_content"), + Index("canvas_items_session", "session_id", "footnote_num"), + ) + + id = Column(Integer, primary_key=True, autoincrement=True) + session_id = Column(String, nullable=False) + type = Column(String, nullable=False) + content = Column(Text, nullable=False) + label = Column(String, nullable=True) + footnote_num = Column(Integer, nullable=False) + created_at = Column(String, nullable=False) # ISO text, matches historical schema diff --git a/web/server/database/alembic.ini b/web/server/database/alembic.ini new file mode 100644 index 0000000..8a4127c --- /dev/null +++ b/web/server/database/alembic.ini @@ -0,0 +1,46 @@ +# Alembic config for the AgentForge chat database (web.server.database). +# Runtime URL is set programmatically — do not hard-code secrets here. + +[alembic] +script_location = web/server/database/migrations +prepend_sys_path = . +version_path_separator = os + +# Overridden at runtime by migrate._alembic_config() +sqlalchemy.url = sqlite:///./data/agentforge_chat.db + +[post_write_hooks] + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/web/server/database/cli.py b/web/server/database/cli.py new file mode 100644 index 0000000..58c6091 --- /dev/null +++ b/web/server/database/cli.py @@ -0,0 +1,142 @@ +"""CLI for AgentForge SQLite migrations (Django ``manage.py migrate`` analogue). + +Usage:: + + python -m web.server.database.cli upgrade-all + python -m web.server.database.cli upgrade --database chat + python -m web.server.database.cli upgrade --database prompt_lab + python -m web.server.database.cli current --database chat + python -m web.server.database.cli applied --database chat + python -m web.server.database.cli history --database chat + python -m web.server.database.cli revision -m "add foo" --database chat --autogenerate +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +from web.server.database import migrate +from web.server.database.migrate import DatabaseName + + +def _project_root() -> Path: + return Path(__file__).resolve().parents[3] + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="agentforge-db", + description="AgentForge SQLite migrations (Alembic) — chat + prompt_lab", + ) + + def _add_common(p: argparse.ArgumentParser) -> None: + p.add_argument( + "--db", + dest="db_path", + default=None, + help="SQLite path override for the selected --database", + ) + p.add_argument( + "--database", + choices=["chat", "prompt_lab"], + default="chat", + help="Which database (default: chat). Ignored by upgrade-all.", + ) + + sub = parser.add_subparsers(dest="command", required=True) + + p_all = sub.add_parser("upgrade-all", help="Upgrade chat + prompt_lab to head") + p_all.add_argument( + "--db", + dest="db_path", + default=None, + help="Optional override for the chat SQLite path only", + ) + + p_up = sub.add_parser("upgrade", help="Apply migrations for --database to head") + _add_common(p_up) + + p_down = sub.add_parser("downgrade", help="Roll back --database") + _add_common(p_down) + p_down.add_argument("revision", help="Target revision (e.g. -1, base)") + + p_cur = sub.add_parser("current", help="Show applied Alembic revision") + _add_common(p_cur) + + p_app = sub.add_parser("applied", help="List schema_migrations rows (filename + revision)") + _add_common(p_app) + + p_hist = sub.add_parser("history", help="List revision ids base → head") + p_hist.add_argument( + "--database", + choices=["chat", "prompt_lab"], + default="chat", + ) + + p_rev = sub.add_parser("revision", help="Create a new migration file") + _add_common(p_rev) + p_rev.add_argument("-m", "--message", required=True) + p_rev.add_argument("--autogenerate", action="store_true") + + args = parser.parse_args(argv) + root = str(_project_root()) + if root not in sys.path: + sys.path.insert(0, root) + + database: DatabaseName = getattr(args, "database", "chat") # type: ignore[assignment] + db_path_arg = getattr(args, "db_path", None) + + if args.command == "upgrade-all": + results = migrate.upgrade_all(chat_db=db_path_arg) + for name, rev in results.items(): + print(f"{name}: {rev or '(empty)'}") + return 0 + + db_path = migrate.resolve_db_path(database, db_path_arg) + + if args.command == "upgrade": + migrate.upgrade(db_path, database=database) + print(f"{database}: {migrate.current(db_path, database=database) or '(empty)'}") + return 0 + + if args.command == "downgrade": + migrate.downgrade(db_path, args.revision, database=database) + print(f"{database}: {migrate.current(db_path, database=database) or '(empty)'}") + return 0 + + if args.command == "current": + print(migrate.current(db_path, database=database) or "(no revision applied)") + return 0 + + if args.command == "applied": + rows = migrate.list_applied(db_path, database=database) + if not rows: + print("(no schema_migrations rows)") + return 0 + for row in rows: + print(f"{row['applied_at']} {row['revision']} {row['filename']}") + return 0 + + if args.command == "history": + for rev in migrate.history(database=database): + print(rev) + return 0 + + if args.command == "revision": + migrate.make_revision( + args.message, + database=database, + autogenerate=args.autogenerate, + db_path=db_path if args.autogenerate else None, + ) + print(f"Revision created under migrations for database={database}") + return 0 + + parser.error(f"unknown command: {args.command}") + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/web/server/database/manager.py b/web/server/database/manager.py index a1349b0..6c70292 100644 --- a/web/server/database/manager.py +++ b/web/server/database/manager.py @@ -4,7 +4,7 @@ - sessionmaker(autocommit=False, autoflush=False) - Context-managed sessions with expunge() before return - Naive local datetimes - - Idempotent create_tables() + - Schema via Alembic (``create_tables`` → ``migrate.upgrade``) """ from __future__ import annotations @@ -14,15 +14,18 @@ from datetime import datetime, timedelta, timezone from pathlib import Path -from sqlalchemy import create_engine, event, func, text +from sqlalchemy import create_engine, event, func from sqlalchemy.orm import sessionmaker from sqlalchemy.pool import NullPool +from web.server.database.migrate import upgrade as alembic_upgrade + from .models import ( Base, ChatMessage, ChatSession, CommandNote, + CommandPolicyOverride, MonitorCheck, MonitorJob, MonitorSnapshot, @@ -75,121 +78,13 @@ def __init__(self, db_path: str | Path) -> None: logger.info("ChatDatabase initialised at %s (journal_mode=DELETE)", self.db_path) def create_tables(self) -> None: - """Create tables if they don't exist (idempotent).""" - Base.metadata.create_all(bind=self.engine) - # Migrate existing databases: add is_incognito if the column doesn't exist yet. - # SQLite does not support IF NOT EXISTS in ALTER TABLE, so we swallow the - # OperationalError that fires when the column is already present. - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_messages ADD COLUMN is_incognito INTEGER NOT NULL DEFAULT 0")) - conn.commit() - logger.info("Migration: added is_incognito column to chat_messages") - except Exception: - pass # Column already exists — no-op - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_messages ADD COLUMN is_volatile INTEGER NOT NULL DEFAULT 0")) - conn.commit() - logger.info("Migration: added is_volatile column to chat_messages") - except Exception: - pass # Column already exists — no-op - # Migrate: add chat_sessions.source (external clients tag themselves so - # the human sidebar can filter them out). Legacy rows default to "web". - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_sessions ADD COLUMN source VARCHAR(32) NOT NULL DEFAULT 'web'")) - conn.commit() - logger.info("Migration: added source column to chat_sessions") - except Exception: - pass # Column already exists — no-op - # Add screenshot_path to monitor_checks if missing - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE monitor_checks ADD COLUMN screenshot_path VARCHAR(500)")) - conn.commit() - logger.info("Migration: added screenshot_path column to monitor_checks") - except Exception: - pass # Column already exists — no-op - # Add structured_selectors to monitor_jobs - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE monitor_jobs ADD COLUMN structured_selectors JSON")) - conn.commit() - logger.info("Migration: added structured_selectors column to monitor_jobs") - except Exception: - pass - # Add structured_content to monitor_snapshots - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE monitor_snapshots ADD COLUMN structured_content JSON")) - conn.commit() - logger.info("Migration: added structured_content column to monitor_snapshots") - except Exception: - pass - # Add structured_diff to monitor_checks - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE monitor_checks ADD COLUMN structured_diff JSON")) - conn.commit() - logger.info("Migration: added structured_diff column to monitor_checks") - except Exception: - pass - # Add created_at to chat_messages - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_messages ADD COLUMN created_at DATETIME DEFAULT (datetime('now'))")) - conn.commit() - logger.info("Migration: added created_at column to chat_messages") - except Exception: - pass # Column already exists - with self.engine.connect() as conn: - try: - conn.execute( - text("ALTER TABLE command_notes ADD COLUMN kind VARCHAR(20) NOT NULL DEFAULT 'tool_calls'") - ) - conn.commit() - logger.info("Migration: added kind column to command_notes") - except Exception: - pass - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE command_notes ADD COLUMN content TEXT")) - conn.commit() - logger.info("Migration: added content column to command_notes") - except Exception: - pass - # Add sequence to chat_messages - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_messages ADD COLUMN sequence INTEGER NOT NULL DEFAULT 0")) - conn.commit() - logger.info("Migration: added sequence column to chat_messages") - except Exception: - pass # Column already exists - # session_instructions table — created via metadata.create_all above. - # No ALTER TABLE migration needed for new table; just ensure it exists. - - # Token usage columns on chat_sessions - for col_name in ("prompt_tokens", "completion_tokens", "total_tokens"): - with self.engine.connect() as conn: - try: - conn.execute(text(f"ALTER TABLE chat_sessions ADD COLUMN {col_name} INTEGER NOT NULL DEFAULT 0")) - conn.commit() - logger.info("Migration: added %s column to chat_sessions", col_name) - except Exception: - pass # Column already exists - - # Per-session AI provider override - with self.engine.connect() as conn: - try: - conn.execute(text("ALTER TABLE chat_sessions ADD COLUMN provider_override VARCHAR(32)")) - conn.commit() - logger.info("Migration: added provider_override column to chat_sessions") - except Exception: - pass # Column already exists + """Apply Alembic migrations to head (idempotent). - logger.info("Database tables ready") + Replaces the old ``create_all`` + ad-hoc ``ALTER TABLE`` path. + See ``web.server.database.migrate`` and ``agentforge-db`` CLI. + """ + alembic_upgrade(self.db_path) + logger.info("Database schema ready (Alembic head) at %s", self.db_path) def drop_tables(self) -> None: """Drop all tables (destructive — for testing/reset).""" @@ -584,6 +479,49 @@ def delete_messages_from_sequence(self, session_id: str, sequence: int) -> int: session.commit() return count + def get_command_policy_override(self, tool: str) -> dict | None: + """Return the runtime override for *tool*, or None if not set.""" + with self.SessionLocal() as session: + row = session.query(CommandPolicyOverride).filter_by(tool=tool).first() + if not row: + return None + session.expunge(row) + return row.to_dict() + + def upsert_command_policy_override(self, tool: str, data: dict) -> dict: + """Insert or update a runtime command policy override for *tool*.""" + with self.SessionLocal() as session: + row = session.query(CommandPolicyOverride).filter_by(tool=tool).first() + if row: + row.mode = data.get("mode", row.mode) + row.allowed_commands = data.get("allowed_commands", []) + row.allowed_patterns = data.get("allowed_patterns", []) + row.blocked_patterns = data.get("blocked_patterns", []) + row.updated_at = datetime.now() + else: + row = CommandPolicyOverride( + tool=tool, + mode=data.get("mode", "confirm"), + allowed_commands=data.get("allowed_commands", []), + allowed_patterns=data.get("allowed_patterns", []), + blocked_patterns=data.get("blocked_patterns", []), + ) + session.add(row) + session.commit() + session.refresh(row) + session.expunge(row) + return row.to_dict() + + def delete_command_policy_override(self, tool: str | None = None) -> int: + """Delete override(s). Pass *tool* to delete one row, or None for all.""" + with self.SessionLocal() as session: + q = session.query(CommandPolicyOverride) + if tool is not None: + q = q.filter_by(tool=tool) + count = q.delete(synchronize_session=False) + session.commit() + return int(count) + # -- Command Notes --------------------------------------------------------- def create_command_note( diff --git a/web/server/database/migrate.py b/web/server/database/migrate.py new file mode 100644 index 0000000..48527d8 --- /dev/null +++ b/web/server/database/migrate.py @@ -0,0 +1,354 @@ +"""Alembic multi-database migration helpers. + +Databases: + +* **chat** — main SQLite (sessions, tools, monitor, connectors, **canvas**). +* **prompt_lab** — separate SQLite for Prompt Lab history. + +Public API: + +* ``upgrade(db_path, database=...)`` / ``upgrade_all()`` +* ``downgrade``, ``current``, ``history``, ``list_applied`` +* ``make_revision`` + +After every successful upgrade/stamp, applied revisions are logged in the +``schema_migrations`` table (Laravel-style): ``revision``, ``filename``, +``applied_at``. +""" + +from __future__ import annotations + +import logging +import os +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Literal + +import yaml +from alembic import command +from alembic.config import Config +from alembic.runtime.migration import MigrationContext +from alembic.script import ScriptDirectory +from sqlalchemy import create_engine, inspect, text +from sqlalchemy.pool import NullPool + +logger = logging.getLogger(__name__) + +DatabaseName = Literal["chat", "prompt_lab"] + +_PACKAGE_DIR = Path(__file__).resolve().parent +_PROJECT_ROOT = _PACKAGE_DIR.parents[3] + + +@dataclass(frozen=True) +class DatabaseSpec: + name: DatabaseName + alembic_ini: Path + migrations_dir: Path + legacy_marker_table: str + default_rel_path: str + + +SPECS: dict[DatabaseName, DatabaseSpec] = { + "chat": DatabaseSpec( + name="chat", + alembic_ini=_PACKAGE_DIR / "alembic.ini", + migrations_dir=_PACKAGE_DIR / "migrations", + legacy_marker_table="chat_sessions", + default_rel_path="data/agentforge_chat.db", + ), + "prompt_lab": DatabaseSpec( + name="prompt_lab", + alembic_ini=_PACKAGE_DIR.parent / "prompt_lab" / "database" / "alembic.ini", + migrations_dir=_PACKAGE_DIR.parent / "prompt_lab" / "database" / "migrations", + legacy_marker_table="prompt_lab_runs", + default_rel_path="data/prompt_lab.db", + ), +} + +_SCHEMA_MIGRATIONS_DDL = """ +CREATE TABLE IF NOT EXISTS schema_migrations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + revision TEXT NOT NULL UNIQUE, + filename TEXT NOT NULL, + applied_at TEXT NOT NULL +) +""" + + +def _sqlite_url(db_path: str | Path) -> str: + path = Path(db_path).expanduser().resolve() + return f"sqlite:///{path}" + + +def _alembic_config(spec: DatabaseSpec, db_path: str | Path) -> Config: + cfg = Config(str(spec.alembic_ini)) + cfg.set_main_option("script_location", str(spec.migrations_dir)) + cfg.set_main_option("sqlalchemy.url", _sqlite_url(db_path)) + cfg.set_main_option("path_separator", "os") + cfg.set_main_option("prepend_sys_path", str(_PROJECT_ROOT)) + return cfg + + +def _script_directory(spec: DatabaseSpec) -> ScriptDirectory: + cfg = Config(str(spec.alembic_ini)) + cfg.set_main_option("script_location", str(spec.migrations_dir)) + cfg.set_main_option("path_separator", "os") + return ScriptDirectory.from_config(cfg) + + +def _has_legacy_schema(spec: DatabaseSpec, db_path: str | Path) -> bool: + path = Path(db_path).expanduser() + if not path.exists() or path.stat().st_size == 0: + return False + engine = create_engine(_sqlite_url(path), poolclass=NullPool) + try: + tables = set(inspect(engine).get_table_names()) + if spec.legacy_marker_table not in tables: + return False + if "alembic_version" in tables: + with engine.connect() as conn: + row = conn.execute(text("SELECT version_num FROM alembic_version LIMIT 1")).fetchone() + if row and row[0]: + return False + return True + finally: + engine.dispose() + + +def _head_revision(spec: DatabaseSpec) -> str: + script = _script_directory(spec) + heads = script.get_heads() + if not heads: + raise RuntimeError(f"No Alembic heads for database={spec.name!r}") + if len(heads) > 1: + raise RuntimeError(f"Multiple Alembic heads for {spec.name!r} (merge required): {heads}") + return heads[0] + + +def _revision_filename(script: ScriptDirectory, revision: str) -> str: + sc = script.get_revision(revision) + if sc is None or not sc.path: + return f"{revision}.py" + return Path(sc.path).name + + +def _ensure_schema_migrations_table(engine) -> None: + with engine.begin() as conn: + conn.execute(text(_SCHEMA_MIGRATIONS_DDL)) + + +def _record_applied_migrations(spec: DatabaseSpec, db_path: str | Path) -> None: + """Sync ``schema_migrations`` with the current Alembic lineage (filename + revision).""" + path = Path(db_path).expanduser() + if not path.exists(): + return + engine = create_engine(_sqlite_url(path), poolclass=NullPool) + script = _script_directory(spec) + try: + _ensure_schema_migrations_table(engine) + with engine.connect() as conn: + context = MigrationContext.configure(conn) + current = context.get_current_revision() + if current is None: + return + + # All revisions from base up to *current* (inclusive). + applied: list[str] = [] + for sc in script.walk_revisions(base="base", head=current): + applied.append(sc.revision) + applied.reverse() + + now = datetime.now(timezone.utc).replace(tzinfo=None).isoformat(timespec="seconds") + with engine.begin() as conn: + for rev in applied: + filename = _revision_filename(script, rev) + conn.execute( + text( + "INSERT OR IGNORE INTO schema_migrations (revision, filename, applied_at) " + "VALUES (:revision, :filename, :applied_at)" + ), + {"revision": rev, "filename": filename, "applied_at": now}, + ) + logger.info( + "schema_migrations synced for %s (%s): %d revision(s)", + spec.name, + path, + len(applied), + ) + finally: + engine.dispose() + + +def upgrade( + db_path: str | Path, + *, + database: DatabaseName = "chat", + revision: str = "head", +) -> None: + """Bring *database* at *db_path* up to *revision* (default: latest head).""" + spec = SPECS[database] + path = Path(db_path).expanduser() + path.parent.mkdir(parents=True, exist_ok=True) + cfg = _alembic_config(spec, path) + + if _has_legacy_schema(spec, path): + head = _head_revision(spec) + logger.info( + "Legacy %s DB at %s — stamping Alembic head %s (no DDL)", + database, + path, + head, + ) + command.stamp(cfg, head) + command.upgrade(cfg, revision) + else: + logger.info("Alembic upgrade %s → %s on %s", database, revision, path) + command.upgrade(cfg, revision) + + _record_applied_migrations(spec, path) + + +def downgrade( + db_path: str | Path, + revision: str, + *, + database: DatabaseName = "chat", +) -> None: + spec = SPECS[database] + cfg = _alembic_config(spec, db_path) + logger.info("Alembic downgrade %s → %s on %s", database, revision, db_path) + command.downgrade(cfg, revision) + # Rebuild log from remaining lineage (remove rows past current). + _prune_schema_migrations_log(spec, db_path) + + +def _prune_schema_migrations_log(spec: DatabaseSpec, db_path: str | Path) -> None: + path = Path(db_path).expanduser() + if not path.exists(): + return + engine = create_engine(_sqlite_url(path), poolclass=NullPool) + script = _script_directory(spec) + try: + tables = set(inspect(engine).get_table_names()) + if "schema_migrations" not in tables: + return + with engine.connect() as conn: + context = MigrationContext.configure(conn) + current = context.get_current_revision() + keep: set[str] = set() + if current is not None: + for sc in script.walk_revisions(base="base", head=current): + keep.add(sc.revision) + with engine.begin() as conn: + if keep: + # Delete rows not in keep + rows = conn.execute(text("SELECT revision FROM schema_migrations")).fetchall() + for (rev,) in rows: + if rev not in keep: + conn.execute( + text("DELETE FROM schema_migrations WHERE revision = :r"), + {"r": rev}, + ) + else: + conn.execute(text("DELETE FROM schema_migrations")) + finally: + engine.dispose() + + +def current(db_path: str | Path, *, database: DatabaseName = "chat") -> str | None: + path = Path(db_path).expanduser() + if not path.exists(): + return None + engine = create_engine(_sqlite_url(path), poolclass=NullPool) + try: + with engine.connect() as conn: + return MigrationContext.configure(conn).get_current_revision() + finally: + engine.dispose() + + +def history(*, database: DatabaseName = "chat") -> list[str]: + script = _script_directory(SPECS[database]) + revs: list[str] = [] + for sc in script.walk_revisions(): + revs.append(sc.revision) + revs.reverse() + return revs + + +def list_applied(db_path: str | Path, *, database: DatabaseName = "chat") -> list[dict]: + """Return rows from ``schema_migrations`` ordered by application time.""" + path = Path(db_path).expanduser() + if not path.exists(): + return [] + engine = create_engine(_sqlite_url(path), poolclass=NullPool) + try: + tables = set(inspect(engine).get_table_names()) + if "schema_migrations" not in tables: + return [] + with engine.connect() as conn: + rows = conn.execute( + text("SELECT revision, filename, applied_at FROM schema_migrations ORDER BY id ASC") + ).fetchall() + return [{"revision": r[0], "filename": r[1], "applied_at": r[2]} for r in rows] + finally: + engine.dispose() + + +def make_revision( + message: str, + *, + database: DatabaseName = "chat", + autogenerate: bool = False, + db_path: str | Path | None = None, +) -> None: + if autogenerate and db_path is None: + raise ValueError("autogenerate requires db_path") + path = Path(db_path).expanduser() if db_path else Path("/tmp/agentforge_alembic_dummy.db") + cfg = _alembic_config(SPECS[database], path) + command.revision(cfg, message=message, autogenerate=autogenerate) + + +def resolve_db_path(database: DatabaseName, explicit: str | Path | None = None) -> Path: + """Resolve SQLite path from CLI arg, env, or config.yaml.""" + if explicit: + return Path(explicit).expanduser().resolve() + env_key = { + "chat": "AGENTFORGE_CHAT_DB", + "prompt_lab": "AGENTFORGE_PROMPT_LAB_DB", + }[database] + env = os.environ.get(env_key) + if env: + return Path(env).expanduser().resolve() + + rel = SPECS[database].default_rel_path + try: + cfg_path = _PROJECT_ROOT / "config.yaml" + if cfg_path.exists() and database == "chat": + with open(cfg_path) as fh: + cfg = yaml.safe_load(fh) or {} + rel = cfg.get("web", {}).get("database_path", rel) + except Exception: + pass + return (_PROJECT_ROOT / rel).resolve() + + +def upgrade_all( + *, + chat_db: str | Path | None = None, + prompt_lab_db: str | Path | None = None, + include_prompt_lab: bool = True, +) -> dict[str, str | None]: + """Upgrade every known database. Returns map of database → current revision.""" + results: dict[str, str | None] = {} + chat_path = resolve_db_path("chat", chat_db) + upgrade(chat_path, database="chat") + results["chat"] = current(chat_path, database="chat") + + if include_prompt_lab: + pl_path = resolve_db_path("prompt_lab", prompt_lab_db) + upgrade(pl_path, database="prompt_lab") + results["prompt_lab"] = current(pl_path, database="prompt_lab") + return results diff --git a/web/server/database/migrations/env.py b/web/server/database/migrations/env.py new file mode 100644 index 0000000..edce18f --- /dev/null +++ b/web/server/database/migrations/env.py @@ -0,0 +1,64 @@ +"""Alembic environment for the AgentForge chat database. + +DB URL is injected at runtime via ``sqlalchemy.url`` in the Alembic config +(see ``web.server.database.migrate``). Autogenerate compares models to the +live schema under ``target_metadata``. +""" + +from __future__ import annotations + +from logging.config import fileConfig + +from alembic import context +from sqlalchemy import engine_from_config, pool + +import web.server.canvas.models # noqa: F401 +from web.server.database.models import Base + +config = context.config + +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = Base.metadata + + +def run_migrations_offline() -> None: + """Run migrations in 'offline' mode (emit SQL only).""" + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + render_as_batch=True, + ) + + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + """Run migrations against a live connection.""" + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + + with connectable.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata, + render_as_batch=True, + compare_type=True, + ) + + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/web/server/database/migrations/script.py.mako b/web/server/database/migrations/script.py.mako new file mode 100644 index 0000000..d5bc78b --- /dev/null +++ b/web/server/database/migrations/script.py.mako @@ -0,0 +1,28 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from __future__ import annotations + +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision: str = ${repr(up_revision)} +down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)} +branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} +depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/web/server/database/migrations/versions/.gitkeep b/web/server/database/migrations/versions/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/web/server/database/migrations/versions/001_baseline_chat_schema.py b/web/server/database/migrations/versions/001_baseline_chat_schema.py new file mode 100644 index 0000000..ecb9aa6 --- /dev/null +++ b/web/server/database/migrations/versions/001_baseline_chat_schema.py @@ -0,0 +1,32 @@ +"""Baseline: full chat database schema (pre-Alembic CREATE + ALTER state). + +Revision ID: 001_baseline +Revises: +Create Date: 2026-07-16 +""" + +from __future__ import annotations + +from typing import Sequence, Union + +from alembic import op + +from web.server.database.models import Base + +revision: str = "001_baseline" +down_revision: Union[str, Sequence[str], None] = None +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Use model metadata so the baseline always matches models.py + # at the time this revision was authored. + # Future changes must be new revisions. + bind = op.get_bind() + Base.metadata.create_all(bind=bind) + + +def downgrade() -> None: + bind = op.get_bind() + Base.metadata.drop_all(bind=bind) diff --git a/web/server/database/migrations/versions/002_add_canvas_items.py b/web/server/database/migrations/versions/002_add_canvas_items.py new file mode 100644 index 0000000..05a5268 --- /dev/null +++ b/web/server/database/migrations/versions/002_add_canvas_items.py @@ -0,0 +1,44 @@ +"""Add canvas_items table (shared chat SQLite file). + +Revision ID: 002_canvas_items +Revises: 001_baseline +Create Date: 2026-07-16 +""" + +from __future__ import annotations + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "002_canvas_items" +down_revision: Union[str, Sequence[str], None] = "001_baseline" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + bind = op.get_bind() + inspector = sa.inspect(bind) + if "canvas_items" in inspector.get_table_names(): + return + + op.create_table( + "canvas_items", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("session_id", sa.String(), nullable=False), + sa.Column("type", sa.String(), nullable=False), + sa.Column("content", sa.Text(), nullable=False), + sa.Column("label", sa.String(), nullable=True), + sa.Column("footnote_num", sa.Integer(), nullable=False), + sa.Column("created_at", sa.String(), nullable=False), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("session_id", "type", "content", name="uq_canvas_session_type_content"), + ) + op.create_index("canvas_items_session", "canvas_items", ["session_id", "footnote_num"]) + + +def downgrade() -> None: + op.drop_index("canvas_items_session", table_name="canvas_items") + op.drop_table("canvas_items") diff --git a/web/server/database/models.py b/web/server/database/models.py index 833e477..3ccbe3d 100644 --- a/web/server/database/models.py +++ b/web/server/database/models.py @@ -171,6 +171,32 @@ def to_dict(self) -> dict: } +class CommandPolicyOverride(Base): + """Runtime command permission override for shell or ssh tools.""" + + __tablename__ = "command_policy_overrides" + + tool = Column(String(16), primary_key=True) # "shell" | "ssh" + mode = Column(String(16), nullable=False, default="confirm") + allowed_commands = Column(JSON, nullable=False, default=list) + allowed_patterns = Column(JSON, nullable=False, default=list) + blocked_patterns = Column(JSON, nullable=False, default=list) + updated_at = Column(DateTime, default=_now, onupdate=_now, nullable=False) + + def __repr__(self) -> str: + return f"" + + def to_dict(self) -> dict: + return { + "tool": self.tool, + "mode": self.mode, + "allowed_commands": self.allowed_commands or [], + "allowed_patterns": self.allowed_patterns or [], + "blocked_patterns": self.blocked_patterns or [], + "updated_at": self.updated_at.isoformat() if self.updated_at else None, + } + + class CommandNote(Base): """A saved bookmark. Tool calls from a run, or an agent answer.""" diff --git a/web/server/permissions/__init__.py b/web/server/permissions/__init__.py new file mode 100644 index 0000000..9154a85 --- /dev/null +++ b/web/server/permissions/__init__.py @@ -0,0 +1 @@ +"""Command permission REST API package.""" diff --git a/web/server/permissions/api.py b/web/server/permissions/api.py new file mode 100644 index 0000000..b065103 --- /dev/null +++ b/web/server/permissions/api.py @@ -0,0 +1,129 @@ +"""REST API for command permission management. + +Mounted under ``/api/permissions``. Exposes YAML baseline, SQLite runtime +overrides, merged effective policy, and a validate endpoint for the Web UI. +""" + +from __future__ import annotations + +from typing import Literal + +from fastapi import APIRouter, Query +from pydantic import BaseModel, Field + +from agentforge.tools.command_policy import ( + CommandPolicy, + PolicyMode, + ToolName, + evaluate, + load_yaml_policy, + merge_policies, +) +from agentforge.tools.command_policy_store import ( + clear_runtime_override, + get_effective_policy, + get_runtime_override, + set_runtime_override, +) + +router = APIRouter(prefix="/api/permissions", tags=["permissions"]) + +_TOOLS: tuple[ToolName, ...] = ("shell", "ssh") + + +class PolicyPayload(BaseModel): + mode: PolicyMode = "confirm" + allowed_commands: list[str] = Field(default_factory=list) + allowed_patterns: list[str] = Field(default_factory=list) + blocked_patterns: list[str] = Field(default_factory=list) + + +class OverridesPayload(BaseModel): + shell: PolicyPayload | None = None + ssh: PolicyPayload | None = None + + +class ValidateRequest(BaseModel): + tool: ToolName + command: str + # Optional draft override from the Web UI — merged with YAML like a saved override. + policy: PolicyPayload | None = None + + +def _policy_to_dict(policy: CommandPolicy) -> dict: + return { + "mode": policy.mode, + "allowed_commands": list(policy.allowed_commands), + "allowed_patterns": list(policy.allowed_patterns), + "blocked_patterns": list(policy.blocked_patterns), + } + + +def _payload_to_policy(payload: PolicyPayload) -> CommandPolicy: + return CommandPolicy( + mode=payload.mode, + allowed_commands=tuple(payload.allowed_commands), + allowed_patterns=tuple(payload.allowed_patterns), + blocked_patterns=tuple(payload.blocked_patterns), + ) + + +def _tool_policy_bundle(tool: ToolName) -> dict: + yaml_policy = load_yaml_policy(tool) + override = get_runtime_override(tool) + effective = get_effective_policy(tool) + return { + "yaml": _policy_to_dict(yaml_policy), + "override": _policy_to_dict(override) if override is not None else None, + "effective": _policy_to_dict(effective), + } + + +@router.get("/commands") +def get_command_policies() -> dict: + """Return YAML baseline, runtime override, and effective policy per tool.""" + return {tool: _tool_policy_bundle(tool) for tool in _TOOLS} + + +@router.get("/commands/overrides") +def get_command_overrides() -> dict: + """Return runtime overrides only (null when unset).""" + result: dict[str, dict | None] = {} + for tool in _TOOLS: + override = get_runtime_override(tool) + result[tool] = _policy_to_dict(override) if override is not None else None + return result + + +@router.put("/commands/overrides") +def put_command_overrides(body: OverridesPayload) -> dict: + """Upsert runtime overrides for shell and/or ssh.""" + if body.shell is not None: + set_runtime_override("shell", _payload_to_policy(body.shell)) + if body.ssh is not None: + set_runtime_override("ssh", _payload_to_policy(body.ssh)) + return {"ok": True} + + +@router.delete("/commands/overrides") +def delete_command_overrides( + tool: Literal["shell", "ssh"] | None = Query(default=None), +) -> dict: + """Delete one override (``?tool=shell``) or all when *tool* is omitted.""" + deleted = clear_runtime_override(tool) + return {"deleted": deleted} + + +@router.post("/commands/validate") +def validate_command(body: ValidateRequest) -> dict: + """Evaluate a command against the effective policy for *tool*.""" + if body.policy is not None: + policy = merge_policies(load_yaml_policy(body.tool), _payload_to_policy(body.policy)) + else: + policy = get_effective_policy(body.tool) + verdict = evaluate(body.tool, body.command, policy) + return { + "action": verdict.action, + "reason": verdict.reason, + "source": verdict.source, + } diff --git a/web/server/prompt_lab/database/alembic.ini b/web/server/prompt_lab/database/alembic.ini new file mode 100644 index 0000000..c4a3baa --- /dev/null +++ b/web/server/prompt_lab/database/alembic.ini @@ -0,0 +1,40 @@ +# Alembic config for prompt_lab.db +[alembic] +script_location = web/server/prompt_lab/database/migrations +prepend_sys_path = . +version_path_separator = os +sqlalchemy.url = sqlite:///./data/prompt_lab.db + +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/web/server/prompt_lab/database/manager.py b/web/server/prompt_lab/database/manager.py index 30ea7a5..72f01e3 100644 --- a/web/server/prompt_lab/database/manager.py +++ b/web/server/prompt_lab/database/manager.py @@ -10,7 +10,9 @@ from sqlalchemy.orm import sessionmaker from sqlalchemy.pool import NullPool -from .models import Base, PromptLabResult, PromptLabRun +from web.server.database.migrate import upgrade as alembic_upgrade + +from .models import PromptLabResult, PromptLabRun logger = logging.getLogger(__name__) @@ -43,7 +45,9 @@ def __init__(self, db_path: str | Path) -> None: logger.info("PromptLabDatabase initialised at %s", self.db_path) def create_tables(self) -> None: - Base.metadata.create_all(bind=self.engine) + """Apply Prompt Lab Alembic migrations to head.""" + alembic_upgrade(self.db_path, database="prompt_lab") + logger.info("Prompt Lab schema ready (Alembic) at %s", self.db_path) # ── run operations ──────────────────────────────────────────────── diff --git a/web/server/prompt_lab/database/migrations/env.py b/web/server/prompt_lab/database/migrations/env.py new file mode 100644 index 0000000..67a63aa --- /dev/null +++ b/web/server/prompt_lab/database/migrations/env.py @@ -0,0 +1,53 @@ +"""Alembic environment for prompt_lab.db.""" + +from __future__ import annotations + +from logging.config import fileConfig + +from alembic import context +from sqlalchemy import engine_from_config, pool + +from web.server.prompt_lab.database.models import Base + +config = context.config + +if config.config_file_name is not None: + fileConfig(config.config_file_name) + +target_metadata = Base.metadata + + +def run_migrations_offline() -> None: + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, + target_metadata=target_metadata, + literal_binds=True, + dialect_opts={"paramstyle": "named"}, + render_as_batch=True, + ) + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online() -> None: + connectable = engine_from_config( + config.get_section(config.config_ini_section, {}), + prefix="sqlalchemy.", + poolclass=pool.NullPool, + ) + with connectable.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata, + render_as_batch=True, + compare_type=True, + ) + with context.begin_transaction(): + context.run_migrations() + + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() diff --git a/web/server/prompt_lab/database/migrations/script.py.mako b/web/server/prompt_lab/database/migrations/script.py.mako new file mode 100644 index 0000000..af11d01 --- /dev/null +++ b/web/server/prompt_lab/database/migrations/script.py.mako @@ -0,0 +1,27 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from __future__ import annotations + +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +revision: str = ${repr(up_revision)} +down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)} +branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} +depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} + + +def upgrade() -> None: + ${upgrades if upgrades else "pass"} + + +def downgrade() -> None: + ${downgrades if downgrades else "pass"} diff --git a/web/server/prompt_lab/database/migrations/versions/.gitkeep b/web/server/prompt_lab/database/migrations/versions/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/web/server/prompt_lab/database/migrations/versions/001_baseline_prompt_lab.py b/web/server/prompt_lab/database/migrations/versions/001_baseline_prompt_lab.py new file mode 100644 index 0000000..98b2c19 --- /dev/null +++ b/web/server/prompt_lab/database/migrations/versions/001_baseline_prompt_lab.py @@ -0,0 +1,27 @@ +"""Baseline: prompt_lab_runs + prompt_lab_results. + +Revision ID: 001_pl_baseline +Revises: +Create Date: 2026-07-16 +""" + +from __future__ import annotations + +from typing import Sequence, Union + +from alembic import op + +from web.server.prompt_lab.database.models import Base + +revision: str = "001_pl_baseline" +down_revision: Union[str, Sequence[str], None] = None +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + Base.metadata.create_all(bind=op.get_bind()) + + +def downgrade() -> None: + Base.metadata.drop_all(bind=op.get_bind()) diff --git a/web/server/scheduler_service.py b/web/server/scheduler_service.py index 5a1eb73..3061d4c 100644 --- a/web/server/scheduler_service.py +++ b/web/server/scheduler_service.py @@ -23,14 +23,25 @@ import subprocess import time import uuid -from typing import TYPE_CHECKING +from collections.abc import Callable +from typing import TYPE_CHECKING, Any from apscheduler.schedulers.background import BackgroundScheduler from apscheduler.triggers.cron import CronTrigger +from agentforge.tools.command_guard import get_guard + if TYPE_CHECKING: from .database import ChatDatabase +_enqueue_scheduled_command: Callable[..., Any] | None = None +try: + from web.server.queue.dispatch_compat import enqueue_scheduled_command as _enqueue_impl + + _enqueue_scheduled_command = _enqueue_impl +except ImportError: + pass + logger = logging.getLogger(__name__) # Maximum output stored per run (chars). Prevents a single noisy job from @@ -180,8 +191,6 @@ def vet_command(self, command: str) -> dict: Reuses the existing command_guard infrastructure. """ try: - from agentforge.tools.command_guard import get_guard - guard = get_guard() verdict = guard.classify(command) return { @@ -190,9 +199,8 @@ def vet_command(self, command: str) -> dict: "source": guard.last_source, } except Exception as exc: - # If the guard is unavailable, fail open with a warning - logger.warning("Command guard unavailable: %s — allowing command", exc) - return {"safe": True, "verdict": "unknown", "source": "unavailable"} + logger.warning("Command guard unavailable: %s — denying command", exc) + return {"safe": False, "verdict": "unknown", "source": "unavailable"} # ------------------------------------------------------------------ # Job execution (called by APScheduler in a thread) @@ -218,9 +226,9 @@ def _execute_job(self, job_id: str) -> None: # (not inside Docker). The worker gives access to terminal-notifier, # SSH keys, brew, Docker CLI, etc. try: - from web.server.queue.dispatch_compat import enqueue_scheduled_command - - enqueue_scheduled_command(job_id, run.id, job.command) + if _enqueue_scheduled_command is None: + raise RuntimeError("dispatch_compat unavailable") + _enqueue_scheduled_command(job_id, run.id, job.command) logger.info( "Scheduled job %s dispatched to host worker (run %s)", job.label,